PaperlayerView exact revision on arXiv

Source-linked AI summary

Blind Super-Resolution Kernel Estimation using an Internal-GAN

Sefi Bell-Kligler, Assaf Shocher, Michal Irani

arXiv:1909.06581v6cs.CV

TL;DR

Most SR methods assume a fixed ideal downscaling kernel, but real LR images use unknown, image-specific kernels that degrade reconstruction quality. KernelGAN learns the appropriate kernel from the LR image alone by matching patch distributions across scales, and the estimate produces state-of-the-art Blind-SR results when used with existing SR algorithms. The approach is fully unsupervised and requires no additional training data.

  • Problem

    Real LR images rarely follow the fixed ideal-kernel assumption embedded in most SR methods, motivating Blind SR with an unknown downscaling kernel.

  • Method

    KernelGAN is an image-specific Internal-GAN whose generator downscales the LR image while matching its patch distribution across scales, using a deep linear network.

  • Results

    KernelGAN’s estimated kernel, plugged into existing SR algorithms, leads to state-of-the-art Blind-SR results by a large margin.

  • Takeaways & Limitations

    The method estimates an image-specific SR kernel from the input image alone, supporting real-world SR without prior training examples.

  • Takeaways & Limitations

    A non-linear generator can satisfy patch-distribution matching with physically unwanted outputs that lack a valid downscaling relation, so the generator must be constrained to be linear.

Abstract

from arXiv · show

Super resolution (SR) methods typically assume that the low-resolution (LR) image was downscaled from the unknown high-resolution (HR) image by a fixed 'ideal' downscaling kernel (e.g. Bicubic downscaling). However, this is rarely the case in real LR images, in contrast to synthetically generated SR datasets. When the assumed downscaling kernel deviates from the true one, the performance of SR methods significantly deteriorates. This gave rise to Blind-SR - namely, SR when the downscaling kernel ("SR-kernel") is unknown. It was further shown that the true SR-kernel is the one that maximizes the recurrence of patches across scales of the LR image. In this paper we show how this powerful cross-scale recurrence property can be realized using Deep Internal Learning. We introduce "KernelGAN", an image-specific Internal-GAN, which trains solely on the LR test image at test time, and learns its internal distribution of patches. Its Generator is trained to produce a downscaled version of the LR test image, such that its Discriminator cannot distinguish between the patch distribution of the downscaled image, and the patch distribution of the original LR image. The Generator, once trained, constitutes the downscaling operation with the correct image-specific SR-kernel. KernelGAN is fully unsupervised, requires no training data other than the input image itself, and leads to state-of-the-art results in Blind-SR when plugged into existing SR algorithms.

1 Introduction

Real LR images usually arise from image-specific, non-ideal downscaling kernels, making fixed-kernel SR methods unreliable. KernelGAN estimates the appropriate kernel from the LR image itself by preserving cross-scale patch distributions through an internal GAN.

  • Motivation: Real LR images often violate the fixed ideal-kernel assumption used by most SR methods, causing poor performance on non-ideal inputs.Synthetic datasets commonly use Bicubic downscaling, whereas real kernels vary with sensor optics and camera motion.
  • Blind SR: Blind SR addresses SR when the downscaling kernel is unknown, and the correct kernel maximizes patch similarity across LR image scales.Cross-scale patch recurrence is image-specific and unsupervised, requiring no prior examples.
  • KernelGAN: KernelGAN is an image-specific Internal-GAN that trains only on the LR test image to estimate the SR kernel preserving patch distributions across scales.Its generator creates a downscaled image whose patches are intended to be indistinguishable from patches of the original LR image.
  • KernelGAN: The trained generator represents the image-specific downscaling operation and uses a deep linear architecture because downscaling is convolution and subsampling.The paper reports that a deep linear network is dramatically superior to a single strided convolution for optimization.
  • Outcome: KernelGAN is fully unsupervised, dataset-invariant, and achieves state-of-the-art Blind-SR results when its estimated kernel is plugged into existing SR algorithms.The method requires no training data beyond the input image and is intended for real-world SR in the wild.

2 Overview of the Approach

KernelGAN estimates an image-specific SR kernel by matching patch distributions between the LR image and its learned downscaled version. It uses an Internal-GAN trained solely on the input image, with a generator that downscales and a discriminator that distinguishes real from generated patches.

  • KernelGAN seeks a downscaled LR image whose patch distribution matches that of the original LR image.This objective targets the kernel that best preserves cross-scale patch statistics.
  • An internal GAN learns the image-specific distribution of patches from a single input image.Unlike conventional GAN training, the examples come from one image rather than a large image dataset.
  • The fully convolutional generator learns to downscale ILR so its patches are indistinguishable from patches of ILR to the discriminator.Both networks operate on patches rather than the whole image.
  • The discriminator produces a D-map whose pixels indicate whether surrounding patches resemble the original distribution, using real and generated crops with one/zero label maps.It alternates between real crops from ILR and fake crops from the generator output.
  • KernelGAN uses an LSGAN variant with an L1 norm and a regularized generator–discriminator objective.The regularization term constrains the downscaling SR kernel.

3 Discriminator

The discriminator learns the LR image’s patch distribution by classifying local crops as real or generated. Its fully convolutional design produces a spatial D-map, with each location corresponding to a local patch.

  • The discriminator learns the patch distribution of ILR from real input crops and fake crops output by the generator.Its task is to distinguish patches belonging to the input-image distribution from generated patches.
  • A 32×32 input crop produces a 32×32 map with values in [0,1].The architecture consists of a 7×7 convolution followed by six 1×1 convolutions with normalization, ReLU, and Sigmoid activations.
  • A fully convolutional patch discriminator uses no pooling or strides, giving it a 7×7 receptive field and patch-level predictions.Each D-map location corresponds to one input patch.

4 Deep Linear Generator = The downscaling SR-Kernel

KernelGAN uses a deep linear generator to represent the downscaling operation while improving optimization over a single linear layer. The learned network can be collapsed into an explicit, regularized SR kernel and adapted across scale factors.

  • 4.1 Deep Linear Generator: The generator is a linear downscaling model, but a deep architecture is used because a single strided convolution empirically fails to converge reliably to the correct solution.The stated explanation concerns the non-convex loss induced by the discriminator.
  • 4 Deep Linear Generator = The downscaling SR-Kernel: A nonlinear generator could produce valid-looking patches without preserving the required physical downscaling relation.The paper gives tiling input patches as an unwanted example.
  • 4 Deep Linear Generator = The downscaling SR-Kernel: Deep linear networks retain the expressiveness of a single linear layer while offering different optimization behavior.They consist of sequential linear layers without activations.
  • 4.1 Deep Linear Generator: The generator uses five hidden convolutional layers with 64 channels and filters producing a 13 ×13 receptive field for the SR kernel.The first filters are 7×7, 5×5, and 3×3, followed by 1×1 filters.
  • 4.2 Extracting the explicit kernel: Sequentially convolving the generator’s stride-1 filters extracts a compact explicit SR kernel that can be supplied to SR algorithms.Explicit extraction also permits physically meaningful kernel priors through regularization.
  • 4.2 Extracting the explicit kernel: The regularization term constrains kernel normalization, boundaries, sparsity, and centering using weighted components.The coefficients are α = 0.5, β = 0.5, γ = 5, and δ = 1.
  • 4.2 Extracting the explicit kernel: A ×4 kernel can be analytically obtained from a ×2 kernel by convolving it with its dilated version.This permits multiple scale kernels from one KernelGAN run while avoiding excessive downscaling of small LR images.

5 Experiments and results

The experiments evaluate KernelGAN’s kernel estimation on synthetic non-ideal and real LR imagery, comparing it with prior estimation methods and SR baselines. KernelGAN improves Blind-SR performance, especially when its estimated kernel is supplied to existing SR algorithms, while deep linear generators outperform single-layer generators.

  • Dataset and evaluation: The evaluation uses real LR images and a synthetic DIV2KRK dataset with ground-truth HR images and true SR-kernels.DIV2KRK contains 100 DIV2K validation images blurred and subsampled using independently generated non-ideal anisotropic Gaussian kernels.
  • Dataset and evaluation: The study measures kernel-estimation accuracy and SR performance using visual comparisons plus PSNR and SSIM on DIV2KRK.Two non-blind SR algorithms receive bicubic, ground-truth, KernelGAN, and alternative estimated kernels for comparison.
  • Ablation: 3.8dB and 1.6dB: deep linear generators outperform single-layer generators for ×2 and ×4 super-resolution on DIV2KRK.Kernel examples also support the superiority of the deep linear network.
  • Results: 1dB and 0.47dB: KernelGAN with ZSSR outperforms state-of-the-art SR results at scales ×2 and ×4, respectively.When kernels deviate from bicubic, standard SR methods tend to produce blurry results, while Blind-SR methods can oversmooth patterns and oversharpen edges.
  • Results: Using image-specific SR-kernels substantially changes SR outcomes, and KernelGAN outperforms prior estimation in both SR performance and visual similarity to the ground-truth kernel.Figure 5 emphasizes that small visible kernel differences can correspond to large PSNR differences.
  • Runtime: Test-time training takes 61 seconds per image on a Tesla V-100 or 102 seconds on a Tesla K-80, independent of image size and scale factor.The trained generator directly contains the resulting SR-kernel, so there is no separate inference step.

6 Conclusion

The paper estimates an image-specific SR-kernel from the LR image alone using an internal GAN trained on the input image’s patch distribution. Supplying this kernel to existing non-blind SR methods produces state-of-the-art Blind-SR results by a large margin.

  • Conclusion: KernelGAN estimates an image-specific SR-kernel from the LR image alone through an internal GAN trained without prior examples.The method learns the input image’s internal patch distribution during test time.
  • Conclusion: When provided to existing off-the-shelf non-blind SR algorithms, KernelGAN’s kernel estimation yields state-of-the-art SR results by a large margin.The conclusion reports both visual and quantitative improvements.
Loading 1909.06581v6…