Source-linked AI summary
"Zero-Shot" Super-Resolution using Deep Internal Learning
Assaf Shocher, Nadav Cohen, Michal Irani
TL;DR
Supervised SR methods are limited by fixed training assumptions that poorly match real images with varying, unknown acquisition conditions. ZSSR trains a small CNN at test time on internal examples from each LR image, and substantially outperforms state-of-the-art SR methods in real-world non-ideal settings. The method is presented as the first unsupervised CNN-based SR approach.
Problem
Supervised SR methods degrade when LR images differ from the fixed acquisition conditions, kernels, and artifact-free data used for training.
Method
ZSSR trains a small image-specific CNN at test time on internal HR-LR examples generated solely from the LR input image.
Results
ZSSR substantially outperforms state-of-the-art SR methods on real-world non-ideal images and is competitive on ideal benchmark datasets.
Takeaways & Limitations
ZSSR enables unsupervised CNN-based SR that adapts to unknown, varying acquisition settings without external examples or prior training.
Takeaways & Limitations
On ideal bicubic images, ZSSR remains below state-of-the-art and may benefit from combining internal and external learning.
Abstract
from arXiv · showhide
Deep Learning has led to a dramatic leap in Super-Resolution (SR) performance in the past few years. However, being supervised, these SR methods are restricted to specific training data, where the acquisition of the low-resolution (LR) images from their high-resolution (HR) counterparts is predetermined (e.g., bicubic downscaling), without any distracting artifacts (e.g., sensor noise, image compression, non-ideal PSF, etc). Real LR images, however, rarely obey these restrictions, resulting in poor SR results by SotA (State of the Art) methods. In this paper we introduce "Zero-Shot" SR, which exploits the power of Deep Learning, but does not rely on prior training. We exploit the internal recurrence of information inside a single image, and train a small image-specific CNN at test time, on examples extracted solely from the input image itself. As such, it can adapt itself to different settings per image. This allows to perform SR of real old photos, noisy images, biological data, and other images where the acquisition process is unknown or non-ideal. On such images, our method outperforms SotA CNN-based SR methods, as well as previous unsupervised SR methods. To the best of our knowledge, this is the first unsupervised CNN-based SR method.
1. Introduction
Supervised SR methods perform well under fixed, ideal acquisition conditions but often degrade on real images with unknown or non-ideal degradations. ZSSR addresses this by training a small CNN at test time on internal examples from the input image, adapting SR to each image.
- Motivation: Supervised SR methods deteriorate when real LR images depart from predefined kernels, scaling factors, and artifact-free imaging conditions.Typical training assumes bicubic downscaling, fixed ×2–×4 scaling, and no sensor noise, non-ideal PSF, or compression artifacts.
- ZSSR: ZSSR trains a small image-specific CNN at test time using examples extracted solely from the LR input image.This uses internal self-supervision rather than external examples or prior training.
- ZSSR: ZSSR exploits recurring information within a single image to infer complex HR-LR relations without the patch-size and Euclidean-similarity restrictions of earlier unsupervised methods.The learned relations are applied to the LR input to produce the HR output.
- Results: ZSSR is competitive on ideal benchmark datasets and surpasses state-of-the-art supervised SR by a large margin on non-ideal images.The paper states that training and testing the image-specific network takes runtime comparable to testing SotA supervised CNNs.
- Scope: ZSSR adapts to known or unknown imaging conditions and supports diverse images and data types without pretraining or large computational resources.The introduction also claims application to arbitrary output size and, theoretically, arbitrary aspect ratio.
2. The Power of Internal Image Statistics
Natural images contain recurring patches within and across scales, giving image-specific statistics strong predictive power. Internal recurrence can recover details unavailable in external image databases, including tiny structures in heavily undersampled regions.
- Internal recurrence: Small patches recur many times within a natural image, both at the same scale and across different scales.This property was empirically verified across hundreds of natural images and reported for almost any small patch in almost any natural image.
- Image-specific detail: Internal evidence can recover tiny structures when matching evidence appears elsewhere in the same image at a different location and scale.In the cited example, internal SR reconstructs tiny balcony handrails that externally trained state-of-the-art methods fail to recover.
- Internal statistics: Internal patch entropy is lower than external patch entropy, so image-specific statistics can provide stronger predictive power than statistics from general image collections.The preference for internal statistics becomes particularly strong as uncertainty and image degradation increase.
3. Image-Specific CNN
The image-specific CNN is trained at test time on internally generated examples from the input image, then applied to that image for super-resolution. Its compact, fully convolutional design supports flexible, image-specific adaptation, with runtime and quality trade-offs governed by the scale strategy.
- Image-Specific CNN: Given a test image I, the CNN learns to reconstruct I from a downscaled version I↓s, then applies I as input to produce I↑s.The fully convolutional network can process images of different sizes.
- Image-Specific CNN: Training examples are generated by repeatedly downscaling the test image into HR fathers and LR sons, then expanding them with rotations and mirror reflections.The geometric transformations add ×8 more image-specific training examples.
- Image-Specific CNN: Gradual super-resolution applies intermediate scale factors and adds generated outputs and transformed versions to the growing training set.This strategy improves robustness and enables large scale factors from very small LR images.
- Architecture & Optimization: A smaller network suffices because LR-HR relations within one image are less diverse than those across a large external image collection.The method uses an 8-hidden-layer, 64-channel-per-layer fully convolutional network with ReLU activations and residual learning.
- Architecture & Optimization: 54 sec is the average runtime for one scale-factor increase, while six intermediate scale factors typically raise PSNR by ∼0.2dB and runtime to ∼5min per image.The authors describe this as a user-selectable trade-off between output quality and runtime.
- Architecture & Optimization: Beyond 800 × 800 images, the proposed network is faster than EDSR+, whose runtime grows quadratically with image size.On the same platform, EDSR+ takes ∼20 sec for SR×2 on a 200×200 image and 5min for an 800 × 800 image.
- Adapting to the Test Image: The network can adapt at test time to the test image’s degradation settings, including the downscaling kernel, SR scale factor, and gradual-scale schedule.Adding small Gaussian noise during training improves performance across several degradation types, including sensor noise and JPEG artifacts.
- Adapting to the Test Image: +1dB improvement over EDSR+ is obtained on non-ideal kernels using rough kernel estimates computed from the test image.Across low-quality images and varied degradation types, the image-specific CNN obtains significantly better SR results than EDSR+.
4. Experiments & Results
ZSSR is competitive with supervised methods on ideal bicubic benchmarks and provides larger gains when LR images contain unknown kernels or degradations. Its internal learning is especially effective for strongly repetitive structures, while some highly recurrent tiny patterns reveal a remaining opportunity for combining internal and external learning.
- 4.1. The ‘Ideal’ Case: ZSSR achieves competitive results on ideal bicubic benchmarks against externally supervised methods and outperforms the unsupervised SelfExSR by a large margin.It is significantly better than SRCNN and sometimes comparable to or better than VDSR.
- 4.1. The ‘Ideal’ Case: In images with very strong internal repetitive structures, ZSSR tends to surpass VDSR and sometimes EDSR+, even under ideal bicubic downscaling.The comparison indicates that internal recurrence can benefit selected image regions beyond externally learned information.
- 4.1. The ‘Ideal’ Case: ZSSR is advantageous mainly in areas with high information recurrence, especially extremely small patterns that have larger examples elsewhere in the image.The paper highlights tiny building windows as an example of cross-location, cross-scale recurrence.
- 4.2. The ‘Non-ideal’ Case: Aliasing from a δ downscaling kernel is another non-ideal case that SotA methods do not handle well.The paper identifies the δ kernel as a special case of a non-ideal kernel.
- 4.2. The ‘Non-ideal’ Case: For unknown noise or JPEG degradation, ZSSR remains robust, whereas supervised SotA methods can degrade enough that bicubic interpolation outperforms them.The tested degradations are Gaussian noise, speckle noise, and JPEG compression, with SR×2 applied without knowing the degradation type.
5. Conclusion
The paper concludes that Zero-Shot SR uses an image-specific CNN trained on internal test-image examples, enabling adaptation to changing, non-ideal acquisition settings. In such settings, it substantially outperforms state-of-the-art SR methods and is presented as the first unsupervised CNN-based SR method.
- Zero-Shot SR trains a small image-specific CNN at test time using internal examples extracted solely from the LR test image.
- The method targets real-world images whose acquisition process is non-ideal, unknown, and varies between images.
- In real-world non-ideal settings, the method substantially outperforms state-of-the-art SR methods qualitatively and quantitatively.
- The authors identify Zero-Shot SR as the first unsupervised CNN-based SR method.