PaperlayerView exact revision on arXiv

Source-linked AI summary

Seven ways to improve example-based single image super resolution

Radu Timofte, Rasmus Rothe, Luc Van Gool

arXiv:1511.02228v1cs.CV

TL;DR

The paper addresses how to improve example-based single-image super-resolution across established methods and benchmarks. It presents seven techniques and combines five into Improved A+, achieving 0.4dB to 0.9dB gains over state-of-the-art methods while maintaining efficient runtime.

  • Problem

    The paper examines how example-based single-image super-resolution methods can be improved across multiple established methods and standard benchmarks.

  • Method

    The authors present seven improvement techniques and combine augmentation, hierarchical search, context reasoning, cascading, and enhanced prediction into Improved A+.

  • Results

    0.4dB to 0.9dB PSNR improvements over state-of-the-art methods such as A+ and SRCNN are reported on standard benchmarks.

  • Takeaways & Limitations

    The techniques are broadly applicable to example-based methods, with several providing significant PSNR gains without significant increases in running time.

Abstract

from arXiv · show

In this paper we present seven techniques that everybody should know to improve example-based single image super resolution (SR): 1) augmentation of data, 2) use of large dictionaries with efficient search structures, 3) cascading, 4) image self-similarities, 5) back projection refinement, 6) enhanced prediction by consistency check, and 7) context reasoning. We validate our seven techniques on standard SR benchmarks (i.e. Set5, Set14, B100) and methods (i.e. A+, SRCNN, ANR, Zeyde, Yang) and achieve substantial improvements.The techniques are widely applicable and require no changes or only minor adjustments of the SR methods. Moreover, our Improved A+ (IA) method sets new state-of-the-art results outperforming A+ by up to 0.9dB on average PSNR whilst maintaining a low time complexity.

1. Introduction

The paper addresses the ill-posed task of reconstructing high-resolution detail from a single low-resolution image by improving example-based super-resolution methods. It introduces seven techniques, evaluates them across representative methods and standard benchmarks, and derives Improved A+ (IA).

  • Motivation: Single-image SR reconstructs high-resolution images from single low-resolution inputs despite multiple plausible high-resolution patches.Example-based SR addresses this ambiguity using corresponding low-resolution/high-resolution patch pairs from internal or external images.
  • Contribution: The paper presents seven techniques for improving example-based SR and applies them to A+, ANR, Zeyde, Yang, and SRCNN.The techniques are combined to derive the Improved A+ method.
  • Results: 0.9dB in PSNR improves A+ on Set5 at ×3, while Yang and ANR improve by 0.8dB and Zeyde by 0.7dB.These gains come from combinations of the proposed techniques.
  • Evaluation: The paper evaluates the techniques within a framework covering standard benchmarks and describes the paper's progression from framework and techniques to generality and conclusions.The stated evaluation uses Set5, Set14, and B100, while the paper structure introduces the framework, seven techniques, results, and conclusions.

2. General framework

The experimental framework trains example-based SR methods from paired low- and high-resolution patches and evaluates them on standard datasets. Its A+ baseline uses anchored local ridge regressors, with experiments also examining data augmentation, dictionary scaling, and search efficiency.

  • Framework: Training uses 91 images, YCbCr luminance, bicubic downscaling, and fixed ×3 magnification for comparing the seven techniques.Quantitative and qualitative evaluation uses Set5, Set14, and B100.
  • Datasets: Set5, Set14, and B100 provide test images ranging from five-image and fourteen-image collections to 100 Berkeley Segmentation Dataset scenes.L20 is a separate 20-image high-resolution dataset used for self-similarity experiments.
  • Methods: The study compares sparse-coding, anchored-regression, and CNN-based SR methods, including Yang, Zeyde, ANR, A+, and SRCNN.The methods differ in dictionary learning, sparse representation, anchored regression, or direct CNN mapping from low to high resolution patches.
  • Anchored regression baseline (A+): A+ uses a 3 × 3 LR patch, PCA-projected gradient features, and a larger HR patch within an anchored regression framework.The shared setup uses K-SVD dictionary training and preserves 99% PCA energy in the LR features.
  • Anchored regression baseline (A+): A+ assigns each LR input patch to a nearest anchor and applies the corresponding stored projection matrix to produce an HR patch.Each anchor has a ridge regressor trained on a fixed-size local neighborhood of LR training patches.

3. Proposed methods

The proposed methods improve example-based SR through more training data, larger dictionaries, and faster hierarchical search. Increasing dictionary size and training samples improves performance, while hierarchical search preserves accuracy with lower search cost.

  • 3.1. Augmentation of training data (A): Augmenting Train91 with rotations and flips produces 728 training images without altering their content.The augmentation follows image-classification practice and expands the training image set.
  • 3.1. Augmentation of training data (A): More training samples improve A+ PSNR, from 31.83dB with 5,000 samples to 32.71dB with 50 million on Set5 at ×3.Test-time running time depends on the number of regressors, not the number of training samples.
  • 3.2. Large dictionary and hierarchical search (H): Larger dictionaries improve sparse-coding and anchored methods because additional anchors reduce LR-feature quantization error and ease local regression.For A+, Set5 PSNR rises from 32.17dB with 16 regressors to 32.92dB with 65,536 regressors and 50 million samples.
  • 3.2. Large dictionary and hierarchical search (H): A hierarchical two-layer search clusters anchors around centroids, then searches assigned anchors for each query.The structure uses k-means centroids and assigns each centroid its cN most correlated anchors, with c = 4 in the experiments.

3.3. Back projection (B)

Iterative back projection refines SR outputs by enforcing consistency with the LR input and degradation operators. Its gains depend on the starting method, and reported results assume known degradation operators.

  • 3.3. Back projection (B): Back projection makes HR reconstructions consistent with the LR input and degradation operators such as blur, downscaling, and downsampling.The degradation operators must be known for iterative back-projection approaches.
  • 3.3. Back projection (B): With degradation operators assumed known, back projection improves A+ PSNR by up to 0.06dB depending on the settings.These gains are reported when starting from A+ results.
  • 3.3. Back projection (B): 0.59dB is the largest reported improvement, obtained when back projection starts from Yang, compared with 0.04dB for A+ on Set5 at ×3.The authors attribute the difference to A+ already being 1.18dB better than Yang before refinement.
  • 3.3. Back projection (B): Because degradation operators are generally unknown and estimated imprecisely, the reported back-projection results are an upper bound for practical implementation.The authors describe these results as difficult to reach in practice.

3.4. Cascade of anchored regressors (C)

Cascading repeatedly applies anchored regression to refine the preceding SR output, while retaining stage-specific models. Performance improves across stages but eventually saturates, and runtime grows linearly with stage count.

  • 3.4. Cascade of anchored regressors (C): Cascaded A+ applies the same features and settings at every stage but uses models trained separately for each stage.The method is called A+C and uses 50 million training samples.
  • 3.4. Cascade of anchored regressors (C): 33.21dB is reached after four cascade stages, compared with 32.92dB after the first stage on Set5 at ×3.Performance saturates at the fourth stage.
  • 3.4. Cascade of anchored regressors (C): Cascade runtime is linear in the number of stages.The paper also reports the same cascading idea applied to image demosaicing with two stages.
  • 3.4. Cascade of anchored regressors (C): Table 2 reports improvements from both cascading and enhanced prediction for super-resolution PSNR on Set5 at ×3.The table summarizes the combined section-level comparison of these two techniques.

3.5. Enhanced prediction (E)

Enhanced prediction averages SR outputs generated from rotated and flipped versions of the LR image, using transformation consistency. The method improves PSNR, with larger gains when combined with more cascade stages.

  • 3.5. Enhanced prediction (E): Enhanced prediction generates eight transformed LR images, applies SR to each, reverses the transformations, and averages the HR outputs.The transformations are rotations and flips, which should yield identical HR results at pixel level.
  • 3.5. Enhanced prediction (E): On Set5 at ×3, enhanced prediction improves PSNR by 0.05dB for one stage and by more than 0.24dB with four cascade stages.The running time is linear in the number of transformations.
  • 3.5. Enhanced prediction (E): Across SR methods, enhanced prediction gains range from +0.05dB for ANR to +0.25dB for Yang.These improvements are reported in Table 3.

3.6. Self-similarities (S)

Image self-similarities provide internal dictionaries whose usefulness depends on image size and textural complexity. For sufficiently large inputs, internal dictionaries can outperform external ones, while context features offer another route to disambiguating reconstructions.

  • Self-similarities (S): Image self-similarities, or patch redundancy, help discriminate between equally possible high-resolution reconstructions.The paper contrasts internal dictionaries built from the input image with external dictionaries learned from training images.
  • Self-similarities (S): Above 246,000 low-resolution pixels, the internal dictionary improves over the external dictionary on downsized L20 images.The reported comparison depends on input-image size and textural complexity.
  • Context reasoning: Context reasoning uses features from a ×3 downscaled image surrounding each low-resolution patch and clusters them into four groups.Each context image is centered on an LR patch and has dimensions equal to the LR patch size times the scaling factor.

3.8. Improved A+ (IA)

Improved A+ combines augmentation, hierarchical search, context reasoning, cascading, and enhanced prediction to improve the A+ baseline. The combined setup yields a large PSNR gain, while some gains increase computation time.

  • Improved A+ (IA): 0.33dB improvement over A+ on Set5 at ×3 comes from augmentation, 65536 regressors, and hierarchical search without increased running time.The augmented setup uses 50 million training samples.
  • Improved A+ (IA): +0.27dB comes from cascading and another 0.25dB from enhanced prediction, with increased computation time.Context reasoning adds 0.1dB while slightly increasing running time.
  • Improved A+ (IA): Figure 8 summarizes the PSNR gains for the seven proposed ways on Set5 at ×3.The figure presents the improvements associated with the techniques applied to A+.
  • Improved A+ (IA): 0.9dB separates Improved A+ from baseline A+ using five techniques: augmentation, hierarchical search, context reasoning, cascading, and enhanced prediction.Internal dictionaries are possible but considered undesirable because of computational cost.

4. Discussion

The discussion presents the techniques as additive and broadly applicable beyond A+, while benchmark and visual evaluations show improved performance across methods, datasets, and magnification factors. IA generally restores sharper details with fewer artifacts than several baselines.

  • Generality of the seven ways: The techniques are presented as additive, generic, and applicable to other example-based single-image super-resolution methods.The study demonstrates them on five methods, with IA combining five of the seven techniques.
  • Benchmark evaluation: Table 5 reports average PSNR on Set5, Set14, and B100 at ×2, ×3, and ×4 using the same Train91 dataset for training.The comparisons include A+, ANR, Zeyde, SRCNN, bicubic interpolation, and NE+LLE.
  • Qualitative evaluation: IA generally restores sharper details with fewer artifacts than A+ and Zeyde in qualitative comparisons at ×3 and ×4.The text specifically notes improved clarity and sharpness in reconstructed text and facial features.

5. Conclusion

The paper proposes seven techniques for improving example-based super-resolution and combines five into the Improved A+ method. The techniques improve PSNR while preserving the core baseline method and are reported as broadly applicable.

  • Conclusion: The paper proposes seven ways to effectively improve example-based super-resolution.The techniques target performance improvements in example-based SR.
  • Conclusion: Improved A+ combines five techniques and reports 0.4dB to 0.9dB improvements over state-of-the-art methods such as A+ and SRCNN.The method is based on the anchored regressors idea of A+.
  • Conclusion: Augmentation, consistency checks, context reasoning, and iterative back projection boost PSNR without significant increases in running time.Hierarchical organization also enables orders of magnitude more regressors at the same running time.
  • Conclusion: The proposed techniques are generic and require no changes to the core baseline method.The paper reports demonstrations mainly on A+ and additionally on ANR, Yang, Zeyde, and SRCNN.
Loading 1511.02228v1…