PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Networks for Image Super-Resolution with Sparse Prior

Zhaowen Wang, Ding Liu, Jianchao Yang, Wei Han, Thomas Huang

arXiv:1507.08905v4cs.CV

TL;DR

Single-image super-resolution is ill-posed, and generic deep SR models largely omit domain expertise such as image priors and degradation models. The paper builds neural networks around sparse coding and cascades them for flexible scaling. It reports improvements over existing deep and shallow SR models in quantitative and qualitative results.

  • Problem

    Single-image super-resolution is ill-posed, and generic deep SR approaches largely ignore domain expertise such as natural-image priors and image-degradation models.

    Source: §1 · p. 1

  • Method

    The paper turns sparse coding into a neural network trained end to end and cascades multiple networks for arbitrary scaling factors.

    Source: §1 · pp. 1–2

  • Results

    The proposed model makes considerable quantitative and qualitative improvements over existing deep and shallow SR models.

    Source: §7 · p. 8

  • Takeaways & Limitations

    Sparse-coding domain knowledge can benefit training speed and model compactness, while a cascade improves scaling flexibility and robustness to artifacts.

    Source: §7 · p. 8

Abstract

from arXiv · show

Deep learning techniques have been successfully applied in many areas of computer vision, including low-level image restoration problems. For image super-resolution, several models based on deep neural networks have been recently proposed and attained superior performance that overshadows all previous handcrafted models. The question then arises whether large-capacity and data-driven models have become the dominant solution to the ill-posed super-resolution problem. In this paper, we argue that domain expertise represented by the conventional sparse coding model is still valuable, and it can be combined with the key ingredients of deep learning to achieve further improved results. We show that a sparse coding model particularly designed for super-resolution can be incarnated as a neural network, and trained in a cascaded structure from end to end. The interpretation of the network based on sparse coding leads to much more efficient and effective training, as well as a reduced model size. Our model is evaluated on a wide range of images, and shows clear advantage over existing state-of-the-art methods in terms of both restoration accuracy and human subjective quality.

1. Introduction

Single-image super-resolution is ill-posed because an LR image contains fewer known values than the unknowns in its HR counterpart, and existing techniques remain unsatisfactory for many applications. This paper combines sparse-coding expertise with deep learning, proposing a sparse-coding network and a cascade intended to improve restoration and scaling flexibility.

1. Introduction

  • Single-image super-resolution infers missing high-frequency content from an LR input, an ill-posed task whose techniques remain unsatisfactory for many real applications.Known LR variables greatly outnumber the unknowns in the HR image.Source: §1 · p. 1
  • Prior SR approaches include analytical priors, edge and gradient statistics, and patch-based sparse combinations of dictionary atoms learned from external databases or similar LR-image examples.Deep networks had also been proposed to learn LR-to-HR mappings end to end.Source: §1 · p. 1
  • Earlier deep SR networks use generic architectures that learn from data while largely ignoring domain expertise such as natural-image priors and image-degradation models.The paper frames the open question as whether such expertise can improve deep architectures or deep learning can improve handcrafted models.Source: §1 · p. 1
  • The proposed sparse-coding network combines domain expertise with deep learning and improves recovery accuracy and human perception over a generic CNN while keeping the model compact.Interpreting each layer physically also provides a principled parameter initialization that improves optimization speed and quality.Source: §1 · pp. 1–2
  • A cascade of sparse-coding networks supports arbitrary scaling factors and reduces artifacts at large factors, with end-to-end training using a multi-scale cost function.A single network performs super-resolution at only one particular scaling factor.Source: §1 · p. 2
  • The paper also conducts a subjective evaluation of several recent state-of-the-art methods.The stated contributions additionally include combining sparse coding with deep learning for faster training and smaller models.Source: §1 · p. 2

2. Related Work

Sparse-coding SR represents paired LR and HR patches with dictionaries and shared sparse coefficients. LISTA networks approximate sparse codes through recurrent updates, with parameters learned from data.

2.1. Image SR Using Sparse Coding

  • Paired LR and HR patches are represented using dictionaries and can share sparse coefficients when the dictionaries are properly defined, allowing an LR patch to recover its HR counterpart.The dictionary pair can be learned jointly with sparse-code inference or through bi-level optimization.Source: §2.1 · p. 2

2.2. Network Implementation of Sparse Coding

  • A LISTA network approximates the sparse code of input y through recurrent stages, with its linear weights and shrinkage thresholds learned from data.The illustrated network has two time-unfolded recurrent stages.Source: §2.2 · p. 2, Figure 1 · p. 2
  • LISTA learns network parameters from training data by back-propagation, unlike ISTA's analytical relationship between network and sparse-coding parameters.This learning yields a good approximation of the underlying sparse code within a fixed number of recurrent stages.Source: §2.2 · p. 2

3. Sparse Coding based Network for Image SR

The SCN turns the sparse-coding super-resolution pipeline into a trainable neural network whose layers correspond to its processing steps. Its sparse-coding structure supports end-to-end optimization and principled initialization, distinguishing it from generic CNNs.

3. Sparse Coding based Network for Image SR

  • The SCN maps bicubic-upscaled LR images to HR images through patch extraction, LISTA sparse coding, HR patch recovery, and patch combination.LISTA uses k recurrent stages to obtain each patch’s sparse code.Source: §3 · p. 2, Figure 2 · p. 3
  • The adjustable-threshold activation is rewritten using two linear scaling layers around a unit-threshold neuron, allowing tunable parameters to reside in the linear layers.The scaling layers use diagonal weights defined by the thresholds and their element-wise reciprocals.Source: §3 · p. 3
  • The final patch-combination layer weights and averages overlapping recovered patches to form the HR image.The recovered HR patch is produced by multiplying the sparse code by the HR dictionary Dx.Source: §3 · p. 3
  • Merging adjacent linear layers reduces computation and redundant parameters, while patch extraction H and patch combination G remain separate because they use additional nonlinear normalization.The architecture has five trainable layers: convolutional H and G, plus three linear layers.Source: §3 · p. 3

Advantages over Previous Models

  • The sparse-coding-derived network jointly optimizes its layer parameters end to end, unlike the prior sparse-coding method’s individually optimized or manually designed variables.With parameters set from dictionaries learned in, SCN can reproduce almost the same results; after training, it learns a more complex regression function.Source: Advantages over Previous Models · p. 3
  • SCN’s LISTA sub-network enforces a sparse representation prior, unlike the generic ReLU mapping used by the CNN in.The sparse-coding interpretation supports more principled filter initialization, faster training, and a smaller model than a vanilla CNN.Source: Advantages over Previous Models · pp. 3–4

4. Network Cascade for Scalable SR

A cascade of small-scale SCNs is proposed to support scalable super-resolution. For large factors, the cascade can outperform a single large-factor SCN, and its stages can be trained using multi-scale supervision.

4. Network Cascade for Scalable SR

  • Repeatedly applying fixed-scale SCNs enables arbitrary scaling factors, addressing the need for separate fixed-factor models.Repeated upscaling can accumulate errors in other settings; the authors report reduced artifacts for large scaling factors with their cascade.Source: §4 · p. 4
  • For ×4 super-resolution, two cascaded SCN×2 models produce a better final result than a single SCN×4 in the illustrated example.The cascade’s second-stage input is sharper and contains fewer artifacts than the bicubic×4 input to the single model.Source: §4 · p. 4, Figure 3 · p. 4
  • Cascading small-factor SCNs can outperform a single large-factor SCN, especially when the target scaling factor exceeds 2.Each stage progressively recovers information lost in bicubic interpolation and compensates for distortion accumulated in earlier stages.Source: §4 · p. 4
  • The cascade connects each SCN’s output to the next SCN’s input through bicubic interpolation and compares each stage’s output with its corresponding ground truth using MSE.The multi-scale objective uses supervision at all scales; the authors train SCNs sequentially with a greedy algorithm and leave back-propagation through bicubic layers for future work.Source: §4 · p. 4, Figure 4 · p. 4

5. Implementation Details

The implementation specifies SCN dimensions, parameter-reduction choices, initialization, and testing procedures. Evaluation uses luminance-only processing, while multi-view testing adds flipped and transposed patches at increased computational cost.

5. Implementation Details

  • The default SCN uses 9×9 LR patches, 100 LR features, dictionary size 128, 5×5 HR patches, and a 5×5 patch-aggregation filter.Training samples are cropped to 56×56 and produce 44×44 outputs.Source: §5 · pp. 4–5
  • Splitting patch extraction and combination into fixed and trainable layers reduces their parameter counts by more than an order without observable performance loss.H uses four trainable filters shifted to 25 fixed positions; G separates fixed pixel alignment from trainable overlap-combination weights.Source: §5 · p. 5
  • Layer-role knowledge guides initialization: H uses Harr-like gradient filters, G uses uniform weights, and the remaining layers are initialized from sparse-coding dictionaries using ISTA [7].The authors empirically set L=C=5 for the initialization equations.Source: §5 · p. 5
  • SCN training usually takes less than one day on the reported workstation, and the authors note that the customized CUDA ConvNet package could be optimized further for SCN.The workstation used 12 Intel Xeon 2.67GHz CPUs and one GTX680 GPU.Source: §5 · p. 5
  • Testing processes only luminance, applies bicubic interpolation to chrominance, and repeatedly upscales by ×2 before any needed downscaling to the target size.Multi-view testing adds flipped and transposed patches to overlapping-patch sampling, improving SR performance at the cost of computation.Source: §5 · p. 5

6. Experiments

The experiments evaluate SCN and CSCN on standard SR benchmarks and subjective comparisons. They report improved accuracy and visual quality, while sparse-coding structure supports faster training and compact models.

6. Experiments

  • Models are trained on 91 images and evaluated on Set5, Set14, and BSD100 using bicubic-downsampled image pairs; training on a larger dataset changes performance by less than 0.1 dB.Source: §6 · p. 5

6.1. Algorithm Analysis

  • The learned first-layer filters retain patterns similar to initial gradient operators while adding structured coefficients that capture richer high-frequency details.Source: §6.1 · p. 5
  • SCN initialized from sparse-coding interpretation converges faster and better than random initialization, and reaches CNN's performance with less than 1% of its back-propagations.A same-size CNN converges much more slowly; the paper reports 8×108 back-propagations for CNN training.Source: §6.1 · pp. 5–6
  • SCN's smallest tested model achieves higher PSNR than CNN-L while using about 20% of its parameters.Reducing dictionary size from n=128 to n=64 does not reduce PSNR too much and significantly reduces model size and computation time.Source: §6.1 · p. 6, Figure 7 · p. 6
  • Increasing recurrent stages from k=1 to k=3 improves performance by less than 0.1 dB, so the paper uses k=1 as a speed-accuracy tradeoff.Source: §6.1 · p. 6
  • SCN×2 matches the scale-specific model at scaling factor 1.5 and performs better at factors 3 and 4; multi-scale training further improves results at factors 3 and 4.Repeated SCN×1.5 upscaling amplifies artifacts, so SCN×2 is used as CSCN's default building block.Source: §6.1 · p. 6, Table 1 · p. 6

6.2. Comparison with State of the Arts

  • CSCN consistently outperforms prior methods in PSNR and SSIM across Set5, Set14, and BSD100, with multi-view testing improving results further.It also performs better than CNN-L despite using a smaller model and training set; its advantage is generally greater at large scaling factors.Source: §6.2 · p. 6, Table 2 · p. 7

6.3. Subjective Evaluation

  • The subjective evaluation ranks CSCN above previous state-of-the-art methods, while CNN is less competitive perceptually than in PSNR comparisons.Scores are estimated from 720 pairwise comparisons by 270 participants using a Bradley-Terry model; ground truth is normalized to 1.Source: §6.3 · p. 8, Figure 10 · p. 8

7. Conclusions

The paper concludes that combining sparse coding with deep networks improves image super-resolution quantitatively and qualitatively. It also identifies applying SCN to other sparse-coding problems and exploring links between low- and high-level vision as future work.

7. Conclusions

  • Combining sparse coding and deep networks improves SR over existing deep and shallow models in quantitative and qualitative evaluations.The paper also highlights training speed, model compactness, and cascaded scaling flexibility and artifact robustness.Source: §7 · p. 8
  • Future work will apply SCN to other problems where sparse coding is useful and explore interaction between low- and high-level vision networks.Source: §7 · p. 8

Related papers

This paper cites

Loading 1507.08905v4…