PaperlayerView exact revision on arXiv

Source-linked AI summary

Meta-Transfer Learning for Zero-Shot Super-Resolution

Jae Woong Soh, Sunwoo Cho, Nam Ik Cho

arXiv:2002.12213v1cs.CV

TL;DR

MZSR addresses CNN super-resolution's limited use of image-internal information and sensitivity to supervised degradation conditions by learning an initial parameter for fast self-supervised adaptation. It combines external and internal samples and achieves competitive results with one gradient update, while remaining fast, flexible, lightweight, and unsupervised at meta-test time.

  • Problem

    CNN-based super-resolution methods rely on external samples, cannot exploit image-internal information, and are limited to supervised degradation conditions.

  • Method

    MZSR combines transfer learning and optimization-based meta-learning to learn an initialization that rapidly adapts through self-supervised learning to image-specific conditions and different blur kernels.

  • Results

    MZSR produces results comparable to other methods with one gradient update and outperforms methods including ZSSR, which requires thousands of iterations.

  • Takeaways & Limitations

    By exploiting both external and internal samples, MZSR offers fast, flexible, lightweight, unsupervised adaptation across image conditions.

  • Takeaways & Limitations

    Multi-scale training performs worse at ×2 than a single-scale model because the more complex task distribution challenges fast-adaptation regions.

Abstract

from arXiv · show

Convolutional neural networks (CNNs) have shown dramatic improvements in single image super-resolution (SISR) by using large-scale external samples. Despite their remarkable performance based on the external dataset, they cannot exploit internal information within a specific image. Another problem is that they are applicable only to the specific condition of data that they are supervised. For instance, the low-resolution (LR) image should be a "bicubic" downsampled noise-free image from a high-resolution (HR) one. To address both issues, zero-shot super-resolution (ZSSR) has been proposed for flexible internal learning. However, they require thousands of gradient updates, i.e., long inference time. In this paper, we present Meta-Transfer Learning for Zero-Shot Super-Resolution (MZSR), which leverages ZSSR. Precisely, it is based on finding a generic initial parameter that is suitable for internal learning. Thus, we can exploit both external and internal information, where one single gradient update can yield quite considerable results. (See Figure 1). With our method, the network can quickly adapt to a given image condition. In this respect, our method can be applied to a large spectrum of image conditions within a fast adaptation process.

1. Introduction

CNN-based SISR methods rely on external data and known degradation conditions, while ZSSR flexibly learns internal image structure but requires thousands of test-time updates. MZSR combines external and internal learning by learning an initialization for fast, flexible adaptation.

  • Motivation: CNN-based SISR methods achieve strong results under known, noise-free bicubic degradation but suffer artifacts and inferior performance under different kernels or noise.Their parameter and memory requirements can also limit real-world use.
  • Motivation: Non-local self-similarity provides an internal image prior used in restoration and can be combined with external information.It represents recurrence within a single image, including across scales.
  • Existing approach: ZSSR adapts to test-image conditions and arbitrary blur kernels by learning internal non-local structure, but requires thousands of gradient updates and lacks large-scale external examples.It can outperform external CNNs where recurrences are salient, yet performs worse in many regions with general patterns.
  • MZSR: MZSR uses meta-transfer learning to learn an effective initial weight for fast adaptation in a zero-shot unsupervised setting.The method is presented as kernel-agnostic and adds meta-training beyond ZSSR’s meta-test-only procedure.
  • MZSR: MZSR first uses external bicubic data, then meta-learns across blur-kernel tasks and performs self-supervised image-specific learning in a few gradient steps.This design exploits both external and internal samples while retaining a lightweight, flexible, unsupervised meta-test process.

2. Related Work

SISR models commonly assume known degradation kernels, whereas real-world conditions vary. Related work addresses this through degradation-aware CNNs, blind or internal learning, and meta-learning across tasks.

  • CNN-based Super-Resolution: The SISR degradation model combines blur, convolution, decimation, and noise, with real scenes containing diverse unknown degradation components.The notation identifies HR and LR images, blur kernel k, convolution, scaling factor s, and white Gaussian noise n.
  • CNN-based Super-Resolution: CNN-based networks perform strongly for bicubic downsampling but suffer under non-bicubic kernels; SRMD adds kernel and noise information, while IKC addresses blind super-resolution.These methods target mismatch between supervised degradation assumptions and broader conditions.
  • MZSR: MZSR uses external training during meta-transfer learning and internal self-supervision during meta-testing for image-specific adaptation to blur kernels.Its scheme proceeds from large-scale training to meta-transfer learning and then meta-test adaptation.
  • Zero-shot Super-Resolution: ZSSR learns image-specific internal structure from a single test image and can flexibly address real-world blur conditions.Its zero-shot procedure complements external-data approaches by adapting to the given image.
  • Meta-Learning: Meta-learning methods include metric-based, memory-based, and optimization-based approaches that improve learning or adaptation across tasks.MAML is identified as an influential optimization-based method that learns a suitable initial state.

3. Preliminary

The preliminary framework describes zero-shot super-resolution as self-supervised learning from a single test image and meta-learning as adaptation across task distributions. In MZSR, blur-kernel settings define the super-resolution tasks.

  • Zero-Shot Super-Resolution: ZSSR creates LR-HR training pairs from a single test image by downsampling the image and uses those pairs to train a CNN at runtime.The original test image serves as HR supervision, while the generated LR son supplies the input.
  • Zero-Shot Super-Resolution: Because ZSSR training depends solely on the test image, it learns internal information specific to that image’s statistics.This is the basis for its zero-shot or self-supervised setting.
  • Meta-Learning: Meta-learning trains a model to adapt across tasks by optimizing a base learner on task-specific training samples and a meta-learner on task test samples.Tasks are sampled from a task distribution during meta-training.
  • MZSR formulation: In MZSR, the input is a degraded LR image and the output is a super-resolved image, while each blur kernel defines a task.Diverse blur kernels therefore form the task distribution for adaptation.

4. Method

MZSR combines large-scale external training, meta-transfer learning across degradation tasks, and image-specific zero-shot adaptation to obtain a transferable initialization for fast super-resolution.

  • Overall scheme: MZSR uses three stages: large-scale training, meta-transfer learning, and meta-test.The meta-test stage performs zero-shot super-resolution on the input image.
  • Large-scale training: Large-scale training uses bicubicly downsampled DIV2K images to learn super-resolution representations and natural-image priors.The network minimizes pixel-wise L1 loss between predictions and ground truth.
  • Meta-transfer learning: Meta-transfer learning trains across diverse blur-kernel tasks to find an initialization where a few gradient updates produce large performance improvements.The task distribution includes isotropic and anisotropic Gaussian kernels with covariance-based random settings.
  • Meta-transfer learning: Unlike MAML, MZSR uses external data during meta-training and internal learning during meta-test to emphasize kernel-agnostic adaptation.This separates task-level learning across synthetic degradation conditions from image-specific adaptation.
  • Meta-transfer learning: During meta-training, task-specific parameters are obtained with gradient descent and the shared parameters are optimized using test error after adaptation.The task-level update uses learning rate α, while meta-optimization uses learning rate β.
  • Meta-test: At meta-test, the input LR image is downsampled with its corresponding kernel to create an internal training pair, followed by a few gradient updates before reconstruction.The final super-resolved image is produced using the updated parameters.

8 Generate task distribution p(T ) with DHR and p(k)

The task-generation procedure initializes meta-testing from the trained model, creates an internally generated LR example from the test image, and performs a small number of adaptation updates.

  • Meta-test inputs: Meta-test takes an LR test image, the meta-transfer-trained parameters, a gradient-update count, and a learning rate as inputs.It outputs a super-resolved image.
  • Internal example generation: The procedure downsamples the LR test image with the corresponding blur kernel to generate an internally synthesized LR image.This creates the image-specific input used for zero-shot adaptation.
  • Adaptation: The model performs n gradient-update steps during meta-test before producing the super-resolved image with the final updated parameters.The method is described as using a few gradient updates rather than the thousands required by standard ZSSR.

5. Experiments

Experiments evaluate MZSR on bicubic and varied-kernel super-resolution benchmarks, measuring PSNR/SSIM and adaptation across gradient updates. MZSR matches ZSSR with one update, performs strongly under non-bicubic conditions, and adapts rapidly despite unseen kernels.

  • Experiments use Set5, BSD100, and Urban100, reporting PSNR and SSIM in the Y channel of YCbCr.
  • On bicubic ×2 benchmarks, MZSR shows comparable results to ZSSR after only one gradient-descent update.
  • The varied-kernel evaluation covers severe aliasing, isotropic Gaussian, anisotropic Gaussian, and Gaussian-plus-bicubic subsampling scenarios.
  • For aliasing with gd 0.2, MZSR remains comparable to other methods after one update and adapts despite never seeing that kernel during training.
  • MZSR outperforms other methods by a significantly large gap for isotropic and anisotropic Gaussian kernels.
  • Real-image evaluations provide visual comparisons only because ground-truth images are unavailable, with comparisons placed in supplementary material.
  • The ablation shows MZSR has the worst initial performance but the best performance after one iteration among compared methods.

6. Discussion

MZSR adapts rapidly to image conditions, achieving strong results after a single gradient update while retaining flexibility for large scaling factors. Its multi-scale model and kernel sensitivity expose important limits, whereas its lightweight design reduces computation relative to ZSSR.

  • 6.1. Number of Gradient Updates: One iteration gives MZSR the best average PSNR among compared configurations on Set5.The initial point performs worst, but MZSR rapidly adapts after one update; other methods can improve more slowly.
  • 6.1. Number of Gradient Updates: MZSR’s visual quality improves sharply after one gradient update, while a pre-trained network changes only modestly.MZSR continues improving with additional updates despite being trained for maximum performance after five steps.
  • 6.2. Multi-scale Models: With scaling factor ×4 and a 30×20 LR son image, MZSR learns internal information and produces plausible results.The example uses blur kernel gd 2.0; the figure identifies Ison and ILR in green boxes.
  • 6.2. Multi-scale Models: Multi-scale training performs worse than the single-scale model at ×2 because the more complex task distribution challenges fast adaptation.The comparison is reported for average PSNR on Set5, with losses shown relative to the single-scale model.
  • 6.2. Multi-scale Models: Larger scaling factors can make Ison too small for sufficient CNN input information, although recurrent internal patterns can still yield plausible images.The limitation is most relevant when the LR son image becomes very small.
  • 6.3. Complexity: MZSR with one gradient update has the shortest comparison time, and ten updates remain comparable to CARN.ZSSR requires thousands of forward and backward passes, making its runtime impractical despite using fewer parameters.

7. Conclusion

MZSR is presented as a fast, flexible, lightweight self-supervised super-resolution method combining external and internal samples. Meta-learning seeks an initialization sensitive to blur-kernel conditions, enabling adaptation within a few gradient updates, while experiments report superior performance to methods such as ZSSR.

  • 7. Conclusion: MZSR combines optimization-based meta-learning with transfer learning to exploit external and internal samples.Transfer learning provides external natural-image priors, while meta-learning targets adaptation across blur-kernel conditions.
  • 7. Conclusion: MZSR quickly adapts to specific image conditions within a few gradient updates.The method is designed to find an initial point sensitive to different blur kernels.
  • 7. Conclusion: Extensive experiments show that MZSR outperforms other methods, including ZSSR, which requires thousands of gradient-descent iterations.The paper also evaluates the method’s complexity.
  • 7. Conclusion: Network architecture, learning strategies, and the multi-scale model remain areas for future improvement.These are identified by the authors as parts of the work that can be improved.

A. Evaluation on Scaling Factor ×4

At scaling factor ×4 with an isotropic Gaussian kernel of width 2.0, MZSR achieves results comparable to other methods even with one gradient update. The multi-scale model again performs worse than the single-scale model at ×2.

  • A. Evaluation on Scaling Factor ×4: Comparable results to other methods are achieved by MZSR with one gradient update at scaling factor ×4.The evaluation uses an isotropic Gaussian kernel with width 2.0.
  • A. Evaluation on Scaling Factor ×4: The multi-scale model performs worse than the single-scale model at scaling factor ×2.This finding is reported alongside the ×4 evaluation.

B. Effects of Kernels on Meta-test Time

Kernel mismatch at meta-test time can severely degrade MZSR’s output. Results become more plausible when the input kernel has a shape similar to the true kernel.

  • B. Effects of Kernels on Meta-test Time: Kernel mismatch severely degrades MZSR output, especially when the input kernel differs greatly from the true kernel.The comparison is illustrated in Figure 6(a) and (b).
  • B. Effects of Kernels on Meta-test Time: An input kernel with a shape similar to the true kernel produces a more plausible result.This case is illustrated in Figure 6(c).

C. Visualization

The section visualizes MZSR results across synthetic blur-kernel scenarios and a real-world image. It also describes comparisons involving different kernels at meta-test time and reports a benchmark table for ×4 scaling.

  • Synthetic blur kernels: Figures 7 and 8 visualize ×2 super-resolution results for synthetic blur-kernel scenarios.Figure 7 uses an anisotropic blur kernel, while Figure 8 uses an isotropic blur kernel with bicubic subsampling.
  • Real-world image: Figure 9 visualizes ×4 super-resolution results on a real-world image.
  • Benchmark results: Table 5 reports average PSNR/SSIM results for ×4 scaling on benchmarks, with parenthetical values indicating the number of gradient updates.The caption states that the best and second-best results are highlighted in red and blue, respectively.
  • Kernel comparisons: Figure 6 compares results obtained with different kernels at meta-test time, including a final result using the true kernel.
Loading 2002.12213v1…