PaperlayerView exact revision on arXiv

Source-linked AI summary

RAISR: Rapid and Accurate Image Super Resolution

Yaniv Romano, John Isidoro, Peyman Milanfar

arXiv:1606.01299v3cs.CV

TL;DR

SISR seeks to recover a high-resolution image from a low-resolution measurement while keeping the learning and runtime complexity low. RAISR learns hashed, patch-adaptive filters and uses structure-aware blending, with sharpening and compression-artifact suppression obtained through suitable training. The paper reports competitive restoration at extremely low computational cost and a sharpener with near-linear-sharpener complexity.

  • Problem

    SISR seeks to recover an unknown high-resolution image from a low-resolution measurement, including when the degradation involves blurring and decimation.

  • Method

    RAISR learns low-complexity filters from image patches, selects them through efficient hashing, and uses structure-aware blending; sharpening is incorporated through training and a related sharpener.

  • Results

    RAISR achieves restoration performance competitive with much more complex state-of-the-art methods despite extremely low computational cost.

  • Takeaways & Limitations

    The framework provides a rapid mechanism for increasing the resolution of previously unseen images while supporting sharpening and suppression of compression artifacts.

Abstract

from arXiv · show

Given an image, we wish to produce an image of larger size with significantly more pixels and higher image quality. This is generally known as the Single Image Super-Resolution (SISR) problem. The idea is that with sufficient training data (corresponding pairs of low and high resolution images) we can learn set of filters (i.e. a mapping) that when applied to given image that is not in the training set, will produce a higher resolution version of it, where the learning is preferably low complexity. In our proposed approach, the run-time is more than one to two orders of magnitude faster than the best competing methods currently available, while producing results comparable or better than state-of-the-art. A closely related topic is image sharpening and contrast enhancement, i.e., improving the visual quality of a blurry image by amplifying the underlying details (a wide range of frequencies). Our approach additionally includes an extremely efficient way to produce an image that is significantly sharper than the input blurry one, without introducing artifacts such as halos and noise amplification. We illustrate how this effective sharpening algorithm, in addition to being of independent interest, can be used as a pre-processing step to induce the learning of more effective upscaling filters with built-in sharpening and contrast enhancement effect.

I. INTRODUCTION

Single-image super-resolution estimates a high-resolution image from a low-resolution input, but existing approaches trade reconstruction quality against computational complexity and artifact risk. RAISR addresses this with efficiently selected learned filters, structure-aware blending, and a related sharpening method.

  • Problem: SISR estimates a high-resolution version of a low-resolution input for applications including zooming, display conversion, and image or video enhancement.The problem is described as estimating an HR image from an LR image, with practical uses such as zoom-in and high-definition display conversion.
  • Prior methods: Linear interpolation methods are simple and low-complexity but use content-independent kernels, limiting reconstruction of complex structures and often causing aliasing and over-smoothing.Nearest-neighbor, bilinear, and bicubic methods are widely used because of their simplicity, but their fixed kernels are not adaptive to image content.
  • Prior methods: Example-based methods learn mappings from synthetically paired LR-HR patches, with adaptive filters or regressors applied according to the input patch.Sparse-representation methods construct filters from learned dictionary atoms, while ANR replaces sparse coding with nearest-atom search and pre-computed projection matrices.
  • Results: RAISR targets high-quality restoration at roughly two orders of magnitude greater speed than leading algorithms, while remaining competitive with more complex state-of-the-art methods.The introduction states that RAISR has similar restoration quality to competing methods while being much faster; the conclusion emphasizes its low computational cost and competitive performance.
  • RAISR: RAISR enhances a cheap initial interpolation using pre-learned filters selected by hashing local gradient statistics, then blends filtered and initial estimates using structure-aware weights.The blending step uses the Census Transform to detect structure changes and reduce artifacts introduced by filtering.
  • Sharpening: The associated sharpener combines DoG filtering with Census-Transform-based structure-aware blending to enhance fine details and overall contrast while avoiding common artifacts.The framework can also learn mappings from compressed LR images to sharpened HR versions, supporting resolution increase, compression-artifact reduction, and contrast enhancement.

II. FIRST STEPS: GLOBAL FILTER LEARNING

The method learns a filter by least-squares fitting patches from interpolated training images to corresponding high-resolution pixels, while reducing learning memory and computation through compact accumulations and parallelization. At run time, a low-resolution image is first interpolated and then filtered with the learned filter.

  • Global filter learning: The filter h is learned by minimizing the Euclidean distance between interpolated training images and desired high-resolution images.Each Ai contains d×d patches from yi, while bi contains corresponding center pixels from xi.
  • Efficient optimization: Memory requirements are on the order of the learned filter size, making the learning stage efficient for large training matrices.The matrix A may be very large, so patches can be sampled rather than using all available patches.
  • Efficient optimization: The learning problem is rewritten using Q = A^T A and V = A^T b, avoiding storage of the full matrix and vector.Q is a small d^2 × d^2 matrix, and V requires less memory than b.
  • Efficient optimization: Chunked products A_j^T A_j and A_j^T b_j can be accumulated independently and parallelized to accelerate computation.The chunks use q rows with q ≪ MN, and the positive semi-definite Q supports an efficient conjugate-gradients solution.
  • Application: At run time, the input low-resolution image is bilinearly interpolated and then processed by the pre-learned filter to produce its high-resolution approximation.The same inexpensive upscaling method is used during learning and application.

III. REFINING THE CHEAP UPSCALING KERNEL: DEALING WITH ALIASING

RAISR refines cheap bilinear upscaling by accounting for spatially varying interpolation and learning filters that suppress aliasing. It then extends this idea with hash-selected locally adaptive filters while retaining low complexity.

  • Spatially varying global filters: Bilinear interpolation applies four possible kernels in 2× upscaling because interpolation weights vary with pixel location.Convolving the bilinear kernel with a learned filter motivates four pixel-type-specific filters.
  • Spatially varying global filters: The learned filters amplify mid-frequencies while suppressing high frequencies containing aliasing components.Their magnitude responses are similar, while their phase responses differ in agreement with shifted interpolation kernels.
  • Spatially varying global filters: At runtime, RAISR bilinearly upscales the LR image, applies the appropriate pre-learned filter for each pixel type, and aggregates filtered patches.For scale factors 3 and 4, the scheme uses 9 and 16 pixel types, respectively.
  • From global to local adaptivity: The global approach improves linear upscaling restoration but remains weaker than state-of-the-art methods because it is not content-adaptive.Its learning stage estimates only a small amount of parameters.
  • From global to local adaptivity: Hashing divides patches into buckets using inexpensive gradient geometry, enabling locally adaptive filters without expensive clustering.Each bucket receives learned filters, and runtime hashing selects the relevant entries for each patch.
  • From global to local adaptivity: The learning process uses low-memory sub-matrix accumulation, which supports large hash tables requiring millions of examples.The stated memory requirement is on the order of the learned filter size.

A. Hash-Table Keys: Local Gradient Statistics (Angle, Strength, Coherence)

RAISR uses local gradient statistics as hash-table keys so learned filters can adapt to image structure. Eigenanalysis supplies angle, strength, and coherence descriptors that generate diverse directional filters.

  • Descriptor construction: Local gradient statistics are computed from neighboring horizontal and vertical gradients around each pixel.The gradients form a matrix whose structure is analyzed locally.
  • Descriptor construction: Eigenanalysis of the weighted gradient matrix yields gradient orientation, strength, and spread information for hashing.The right singular vector gives orientation, while eigenvalues characterize gradient magnitude and distribution.
  • Filter behavior: Higher coherence produces more directional filters, while higher strength increases filter intensity.For high coherence, filters smooth orthogonally to the gradient and sharpen along its direction.
  • Filter behavior: The learned filters for 3× and 4× upscaling are not simple scaled versions of the 2× filters.They incorporate additional information from the corresponding learned descriptors.
  • Descriptor construction: Coherence is a unitless measure ranging from 0 to 1 that combines the two eigenvalues.Strength and coherence help distinguish local structures such as noise, corners, edges, and stripes.
  • Hash-based adaptation: Quantized angle, strength, and coherence values are incorporated into hash keys, allowing filters to adapt to local image conditions.The resulting filter family is intended to handle a variety of situations.

B. Using Patch Symmetry for Nearly-Free 8× More Learning Examples

RAISR augments scarce hash-bucket training data through patch symmetry. Rotations and mirror transformations provide eight related examples while requiring only a negligible additional accumulation cost.

  • Data imbalance: Reliable learning of a 9 × 9 or 11 × 11 filter requires at least 105 patches.With B hash buckets, this implies 105 patches per bucket, which real-world data may not supply evenly.
  • Data imbalance: Hash values are unevenly distributed because horizontal, vertical, and flat image structures occur more frequently than other structures.This makes some buckets common and others uncommon.
  • Symmetry augmentation: Each patch generates eight examples through four 90° rotations and four mirrored 90° rotations.The transformed patches can enter different hash buckets and pixel shifts.
  • Symmetry augmentation: Symmetry augmentation effectively incorporates 8 times as much information for learning each original patch.The approach uses rebucketing to account for transformations such as x-flips, y-flips, and xy-swaps.
  • Efficient implementation: The transformations can be accumulated algebraically rather than applied as actual image transformations to incoming patches.A number of angle buckets divisible by 4 supports the required symmetry mapping.
  • Efficient implementation: The additional accumulation needed for symmetry takes less than 0.1% of learning runtime.The reported measurement used a 3.4GHz 6-Core Xeon processor.

C. Built-in Suppression of Compression Artifacts and Sharpening Effect

RAISR can learn mappings for compressed or sharpened targets, embedding artifact suppression and sharpening into its upscaling filters. The training preprocessing controls the resulting output characteristics.

  • Compression and sharpening: Real-world images may be blurred by unknown kernels, compressed, post-processed, or contaminated by noise.These conditions differ from the commonly used blur-and-decimation degradation model.
  • Compression and sharpening: Learning from compressed LR images to uncompressed HR targets suppresses compression artifacts.The compression quality parameter affects the learned outcome.
  • Compression and sharpening: Aggressive compression such as quality 80 can suppress artifacts but may produce a smoothed result.The paper contrasts this with moderate compression, such as quality 95, which also helps suppress aliasing.
  • Compression and sharpening: Learning from LR images to sharpened HR targets makes the resulting RAISR upscaling sharper as training sharpening increases.Sharpening is built into the pre-learned filters, so no separate runtime sharpening step is required.
  • Combined effect: By preprocessing training data with compression and sharpening, RAISR can map compressed LR inputs to sharpened HR outputs.The learned filters can suppress compression artifacts and amplify the underlying signal.

D. Blending: An Efficient Structure-Preserving Solution

RAISR uses Census Transform-based blending to choose between inexpensive interpolation and learned filtering, preserving structure while enabling selective sharpening or broader contrast enhancement.

  • D. Blending: An Efficient Structure-Preserving Solution: RAISR blends a cheap upscaled image with a filtered image, selecting contributions according to local structure.The blending avoids slower region clustering while combining appropriate outputs pixel-wise.
  • D. Blending: An Efficient Structure-Preserving Solution: The first blending scheme uses a randomness measure to amplify high frequencies in structured regions while retaining interpolated pixels in flat areas.Its weights increase with the Least Connected Component size and support edge-aware blending.
  • D. Blending: An Efficient Structure-Preserving Solution: The Census Transform represents local structure through an 8-bit string of center-versus-neighbor intensity comparisons, with a threshold allowing small variations.It depends on relative ordering rather than absolute intensity values.
  • D. Blending: An Efficient Structure-Preserving Solution: Pre-sharpened high-resolution training targets provide learned filters with built-in detail and contrast enhancement, while both blending variants aim to avoid artifacts and structure deformation.High-frequency-only enhancement preserves the low- and mid-frequency content of conventional SISR, whereas the alternative also changes contrast.
  • D. Blending: An Efficient Structure-Preserving Solution: The second scheme computes Census-Transform Hamming-distance changes between initial and filtered images to enable contrast enhancement while avoiding major structural changes.Larger bit changes produce larger blending weights, allowing a wide range of frequencies to be enhanced.

V. CT-BASED DOG SHARPENER

The CT-based DoG sharpener combines frequency-selective Difference-of-Gaussian filtering with content-aware blending to achieve efficient sharpening while reducing common artifacts.

  • V. CT-BASED DOG SHARPENER: Difference-of-Gaussian filtering subtracts Gaussians with different standard deviations to form a selectable bandpass filter for amplifying frequency ranges.The parameters control blur, amplification, and the passed frequency range.
  • V. CT-BASED DOG SHARPENER: Naive DoG sharpening can amplify noise and create halos, over-sharpening, gradient reversals, and other artifacts because it is not content-adaptive.These issues motivate adding adaptivity without abandoning the computational advantages of linear DoG filtering.
  • V. CT-BASED DOG SHARPENER: The proposed sharpener integrates CT blending with multiple DoG-enhanced versions so the input is locally fused across different frequency bands.A cascade reuses previously filtered images to reduce computation.
  • V. CT-BASED DOG SHARPENER: The method has complexity equivalent to separable linear filtering followed by a pixel-wise weighted average, with weights computed from inexpensive 3 × 3-neighborhood operations.The descriptor uses basic Boolean comparisons and Hamming-distance evaluation.
  • V. CT-BASED DOG SHARPENER: Counting modified Census bits enhances contrast across relatively low frequencies, whereas the randomness measure enhances content only along edges and structures.The two blending mechanisms therefore produce distinct enhancement effects.

VI. EXPERIMENTS

The experiments evaluate RAISR on conventional SISR benchmarks and on real-world compressed images, including contrast-enhanced training targets produced by the CT-based DoG sharpener.

  • VI. EXPERIMENTS: The evaluation covers conventional SISR at 2×, 3×, and 4× upscaling factors against several state-of-the-art algorithms.The study also examines arbitrary compressed images as a real-world scenario.
  • VI. EXPERIMENTS: For compressed-image experiments, RAISR learns filters from compressed low-resolution images to contrast-enhanced high-resolution versions.The high-resolution training images are pre-processed with the proposed CT-based DoG sharpener.

A. Single Image Super-Resolution

On Set5 and Set14, RAISR delivers competitive SISR quality across 2×–4× upscaling while running substantially faster than the evaluated baselines.

  • A. Single Image Super-Resolution: Set5 and Set14 contain 5 and 14 standard images, with bicubic downscaling used to generate test low-resolution inputs.Evaluation uses luminance-channel PSNR and SSIM, where higher values indicate better restoration.
  • A. Single Image Super-Resolution: RAISR has similar restoration quality to fast A+ and outperforms NE+LS, Zeyde et al., GR, and ANR, while 1024-atom A+ and SRCNN perform better at increased computational cost.All baseline methods significantly outperform bicubic interpolation.
  • A. Single Image Super-Resolution: Fig. 13 compares average PSNR and SSIM against average runtime across methods and 2×, 3×, and 4× upscaling factors.Point size represents the standard error of each quality metric.
  • A. Single Image Super-Resolution: RAISR is competitive in quality and much faster, with runtime about one to two orders of magnitude below the baseline methods.The runtime measurement uses a 3.4GHz 6-Core Xeon desktop computer.
  • A. Single Image Super-Resolution: Visual comparisons show RAISR competitively restores edges and fine details in Zebra, Flowers, and Baby images at 2× upscaling.The Butterfly example illustrates sharper outputs from sharpened training filters, with blending reducing resulting artifacts.

B. All in One Enhancement

RAISR combines efficient upscaling with learned sharpening and contrast enhancement, including robustness to compressed, blurred, and noisy inputs. Experiments report competitive visual and quantitative quality with substantially lower runtime than state-of-the-art methods.

  • Upscaling: More than two orders of magnitude faster than A+ and SRCNN, the bilinear-based RAISR reduces runtime while retaining competitive quality.Using bilinear interpolation as the initial upscaler reduces runtime by about a factor of 2 relative to another RAISR version.
  • Contrast enhancement: Pre-processing HR training images with contrast enhancement produces sharper, higher-contrast upscaled results.The visual comparisons report enhanced letters in Newspaper and amplified fine details in Painting images.
  • Compression handling: Learning with compressed LR images reduces compression artifacts without sacrificing sharpness.Compared with training without compression, PSNR improves by 0.43dB on Set5 and 0.34dB on Set14; SSIM improves by 0.017 and 0.012, respectively.
  • Robustness: RAISR reduces artifacts and preserves fine details across compressed, blurred, and noisy images.The reported examples include restored circle details, artifact suppression, and improved robustness for noisy inputs.

VII. CONCLUSION

RAISR is a rapid, learning-based SISR framework that uses locally adaptive hashed filters and blending to reconstruct images while combining aliasing removal, sharpening, and artifact suppression. Its low computational cost supports competitive restoration performance, though hashing and filter learning remain opportunities for improvement.

  • Framework: RAISR learns a mapping from cheaply upscaled LR images to HR images using locally adaptive filters selected by hashing patch geometry.Training groups patches with similar geometry into buckets and learns one filter per bucket.
  • Framework: The CT-based blending step combines cheap and filtered HR estimates to obtain accurate, artifact-free reconstruction at negligible computational cost.Different blending mechanisms support high-frequency-only or broader frequency enhancement.
  • Efficiency: RAISR adds roughly 2–3 linear filters per image, while hashing makes filter selection constant-complexity rather than linear in dictionary size.The approach remains competitive with much more complex state-of-the-art methods.
  • Enhancement: Training with sharpening and compression produces filters that jointly perform de-aliasing, sharpening, and compression-artifact suppression.The learned filters combine smoothing near the kernel center with edge-ridge sharpening behavior.
  • Enhancement: RAISR’s learned filters capture profiles that would be nearly impossible to construct by hand.The paper also introduces a DoG-based sharpener that flexibly enhances a wide range of frequencies while using CT blending to reduce artifacts.
  • Limitations: The hashing mechanism is crucial, and further study of the hash function could improve results.The paper also suggests that efficient-prior regularization may improve the simple least-squares filter solver.

APPENDIX

The appendix describes Iterative Back Projection as a global post-processing step that enforces consistency between the observed LR image and an estimated HR image. The resulting optimization is solved through several gradient-descent iterations and is useful for conventional SISR.

  • Iterative Back Projection reduces artifacts and estimation errors as a global post-processing step.
  • The method demands equivalence between the LR image and the downscaled estimated HR image.
  • Given RAISR’s HR estimate, the procedure minimizes an objective to obtain the desired HR estimation.
  • The optimization is solved by applying several iterations of gradient descent.
  • The post-processing step was found helpful for the conventional SISR process.
Loading 1606.01299v3…