PaperlayerView exact revision on arXiv

Source-linked AI summary

Flow-based Kernel Prior with Application to Blind Super-Resolution

Jingyun Liang, Kai Zhang, Shuhang Gu, Luc Van Gool, Radu Timofte

arXiv:2103.15977v1cs.CVeess.IV

TL;DR

Blind SR kernel estimation is ill-posed, and existing kernel priors do not fully exploit suitable kernel distributions. FKP learns an invertible flow-based kernel prior and optimizes latent variables instead of network parameters, improving kernel estimation and blind SR performance with lower resource use.

  • Problem

    Blind SR must estimate unknown blur kernels, but existing kernel-modeling approaches have limitations and poor kernel estimates seriously degrade HR reconstruction.

  • Method

    FKP learns an invertible mapping between kernel and latent spaces and replaces kernel-modeling modules in Double-DIP and KernelGAN while optimizing latent variables with fixed flow parameters.

  • Results

    FKP improves kernel estimation accuracy and blind SR performance while using fewer parameters, runtime, and memory than existing kernel-estimation approaches.

  • Takeaways & Limitations

    The learned kernel manifold provides reasonable initialization and constrains optimization, improving stability and enabling state-of-the-art blind SR results.

  • Takeaways & Limitations

    The approach remains evaluated within blind SR settings whose kernel estimation is severely ill-posed, including limitations identified for existing methods under downsampling and weak patch distributions.

Abstract

from arXiv · show

Kernel estimation is generally one of the key problems for blind image super-resolution (SR). Recently, Double-DIP proposes to model the kernel via a network architecture prior, while KernelGAN employs the deep linear network and several regularization losses to constrain the kernel space. However, they fail to fully exploit the general SR kernel assumption that anisotropic Gaussian kernels are sufficient for image SR. To address this issue, this paper proposes a normalizing flow-based kernel prior (FKP) for kernel modeling. By learning an invertible mapping between the anisotropic Gaussian kernel distribution and a tractable latent distribution, FKP can be easily used to replace the kernel modeling modules of Double-DIP and KernelGAN. Specifically, FKP optimizes the kernel in the latent space rather than the network parameter space, which allows it to generate reasonable kernel initialization, traverse the learned kernel manifold and improve the optimization stability. Extensive experiments on synthetic and real-world images demonstrate that the proposed FKP can significantly improve the kernel estimation accuracy with less parameters, runtime and memory usage, leading to state-of-the-art blind SR results.

1. Introduction

Blind SR must estimate unknown blur kernels, making kernel estimation a crucial and ill-posed step. FKP addresses limitations of existing kernel priors by learning an invertible kernel–latent mapping and optimizing latent variables.

  • Blind SR estimates the blur kernel alongside the HR image, and kernel errors seriously degrade reconstructed image quality.
  • Existing methods such as Double-DIP and KernelGAN use network-based kernel modeling, but their suitability and stability for blind SR kernel estimation remain limited.
  • FKP learns an invertible mapping between kernel and latent spaces and can replace kernel-modeling modules in existing blind SR methods.
  • Optimizing FKP's latent variable keeps kernels on a learned manifold, enabling reasonable initialization and more stable kernel estimation.
  • FKP uses 143K parameters versus 641K for Double-DIP and 151K for KernelGAN while improving kernel estimation accuracy.

2. Related Work

Prior kernel-estimation methods use image priors, fully connected networks, or patch recurrence, while normalizing flows provide invertible mappings from complex distributions to tractable ones.

  • Double-DIP uses untrained encoder-decoder networks as image priors, while related work also explores fully connected networks as kernel priors.
  • KernelGAN estimates kernels from cross-scale patch recurrence using a deep linear generator and discriminator.
  • Whether fully connected network priors work for blind SR kernel estimation remains open because blind SR is severely ill-posed by downsampling.
  • Normalizing flows are invertible generative models that transform complex data distributions into simple, tractable distributions.

3. Flow-based Kernel Prior

FKP models blur-kernel distributions with a normalizing flow that bijectively maps kernels to latent variables. It is trained by likelihood and used by optimizing latent variables while keeping its parameters fixed.

  • The SR degradation model forms the LR image by blurring the HR image and then downsampling, while blind SR estimates both the HR image and blur kernel.
  • The MAP formulation combines data fidelity with image and kernel priors, and poor kernel estimation severely harms HR image reconstruction.
  • FKP is built from flow blocks containing batch normalization, permutation, and affine transformation layers.
  • FKP defines a bijection between kernel k and latent variable z_k, allowing exact reconstruction through the inverse mapping.
  • After training with negative log-likelihood, FKP samples a latent variable, fixes model parameters, and updates the latent variable using the kernel-estimation loss.
  • Restricting the latent norm to the Gaussian sphere surface avoids optimizing throughout the entire latent space while generating reliable kernels along the learned manifold.

4. Incorporating FKP to Double-DIP

DIP-FKP replaces the untrained kernel network in Double-DIP with FKP, jointly estimating the HR image and kernel through latent-variable optimization. The learned kernel manifold supports stable kernel estimation, although the self-supervised setup has limited SR image reconstruction performance.

  • 4.1. Original Double-DIP: The original Double-DIP optimizes randomly initialized network parameters for both image and kernel estimation using LR reconstruction error.Its kernel prior is an untrained neural network with fixed random inputs.
  • 4.1. Original Double-DIP: Untrained networks can fail to generate reasonable kernels because kernels are small, lack natural-image properties, and blind SR is highly ill-posed.The supplied discussion identifies these characteristics as obstacles to directly applying Double-DIP.
  • 4.2. Proposed DIP-FKP: DIP-FKP jointly estimates the HR image and blur kernel by incorporating FKP into the Double-DIP framework.FKP replaces the DIP-like kernel prior while DIP continues to estimate the HR image.
  • 4.2. Proposed DIP-FKP: Optimizing the latent variable z_k rather than fixed FKP parameters generates kernels that remain on the learned kernel manifold.The LR reconstruction loss back-propagates through the predicted kernel to z_k.
  • 4.2. Proposed DIP-FKP: DIP-FKP can estimate the kernel accurately and stably, but its self-supervised image reconstruction is limited, so USRNet produces the final SR result.The method uses the estimated kernel as input to a non-blind SR model.

5. Incorporating FKP to KernelGAN

KernelGAN-FKP replaces KernelGAN’s deep linear generator with FKP, optimizing a latent variable to generate kernels constrained by the learned kernel manifold. This addresses instability associated with weak patch distributions and removes the need for extra kernel regularization terms.

  • 5.1. Original KernelGAN: KernelGAN relies on recurring cross-scale image patches and matches patch distributions between the LR and re-downscaled LR images.This internal-GAN formulation motivates estimating the blur kernel from a single LR image.
  • 5.1. Original KernelGAN: Original KernelGAN uses a deep linear generator and several kernel regularization terms, but its performance is unstable and hyper-parameter selection is burdensome.The regularizers include bicubic similarity, pixel-sum, boundary, sparsity, and centrality constraints.
  • 5.1. Original KernelGAN: KernelGAN’s instability can arise when weak image patch distributions allow multiple kernels to produce similar re-downscaled images.In that case, the discriminator may not distinguish the patch distributions and can lead to an incorrect kernel.
  • 5.2. Proposed KernelGAN-FKP: KernelGAN-FKP directly generates the blur kernel from z_k and uses it to create a re-downscaled LR image for discriminator training.The latent variable is optimized to fool the discriminator into matching real and fake patch distributions.
  • 5.2. Proposed KernelGAN-FKP: Optimizing z_k traverses the constrained kernel space, producing more stable convergence than original KernelGAN without extra regularization terms.The supplied text attributes this to the constrained optimization space and improved kernel generation quality.

6.1. Experimental Setup

The experiments adopt anisotropic Gaussian kernels, reflecting the common assumption that real SR blur kernels are typically unimodal and Gaussian. FKP is trained unsupervised on randomly generated kernels, while the two application variants use separate optimization settings.

  • Data Preparation: Experiments use anisotropic Gaussian kernels because real LR-image blur kernels are usually unimodal and can typically be modeled by Gaussians.The paper notes that this assumption is widely adopted in blind SR.
  • FKP: FKP is trained unsupervised on randomly generated anisotropic Gaussian kernels for 50,000 Adam iterations.The flow uses five blocks and an FCN depth of three, with hidden-layer dimension 5(s + 1).
  • DIP-FKP: DIP-FKP is optimized for 1,000 Adam iterations with learning rates 0.1 for FKP and 0.005 for DIP.The DIP optimizer uses β1 = 0.9 and β2 = 0.999.
  • KernelGAN-FKP: KernelGAN-FKP trains on random 64 × 64 LR patches using WGAN-GP, with 1,000 iterations for scale 2 and 4,000 for scale 4.The gradient penalty is 0.1 and the batch size is 64.

6.2. Experiments on DIP-FKP

DIP-FKP improves blind SR kernel estimation across quantitative, robustness, efficiency, visual, and optimization evaluations. Its flow-based prior yields more accurate kernels and more stable convergence than competing kernel priors.

  • Quantitative Results: DIP-FKP improves DIP across all datasets and scale factors, with estimated kernels further improving non-blind SR performance.The comparison includes bicubic interpolation, RCAN, DIP, Double-DIP, and a ground-truth-kernel upper bound.
  • Quantitative Results: DIP-FKP significantly outperforms the parametric prior in kernel estimation, producing better image PSNR after non-blind SR.Before non-blind SR, both priors achieve similar image PSNR because image quality depends heavily on DIP.
  • Robustness: Under heavy kernel corruption, DIP-FKP remains comparable, and under image noise it still outperforms competitors despite a moderate performance drop.The image-noise setting adds level-10 noise after blurring and downsampling.
  • Efficiency: DIP-FKP uses 143K parameters, about 280 seconds, and 10.6GB memory, compared with Double-DIP's 641K parameters, 300 seconds, and 11.2GB.These measurements use a Tesla V100 GPU and a 1,024 × 1,024 HR image.
  • Visual Results: DIP-FKP produces kernels close to ground truth and the most visually pleasant SR results on synthetic and real-world images.The comparison is conducted at scale factor 4 against RCAN, DIP, DIP+Softmax, and Double-DIP.
  • Visual Results: DIP-FKP starts from a good kernel initialization and converges better than randomly initialized Double-DIP during optimization.The learned kernel manifold constrains the optimization trajectory.

6.3. Experiments on KernelGAN-FKP

KernelGAN-FKP improves kernel and image reconstruction quality over KernelGAN while reducing generator parameters and maintaining competitive efficiency. It also converges more stably and produces fewer visual artifacts.

  • Quantitative Results: KernelGAN-FKP improves kernel PSNR by 2.83dB and 3.35dB at scale factors 2 and 4, yielding image PSNR gains of 1.1dB and 1.77dB after non-blind SR.The gains are measured against KernelGAN on DIV2K.
  • Robustness: KernelGAN-FKP outperforms other methods by substantial margins under non-Gaussian kernels and noisy images.Its performance under image noise is comparable, while all methods experience performance drops.
  • Efficiency: KernelGAN-FKP reduces generator parameters from 151K to 143K and uses 90 seconds and 1.5GB memory versus KernelGAN's 93 seconds and 1.3GB.Runtime and memory are reported on a Tesla V100 GPU.
  • Visual Results: KernelGAN-FKP generates more accurate blur kernels and fewer artifacts than KernelGAN on synthetic and real-world images.The visual comparison is reported for scale factor 4.
  • Optimization: KernelGAN-FKP converges more stably than KernelGAN, whose intermediate kernels oscillate during optimization.The comparison uses intermediate kernel results during optimization.

6.4. DIP-FKP v.s. KernelGAN-FKP

DIP-FKP generally offers better kernel estimation and more stable convergence for small images, whereas KernelGAN-FKP uses much less memory but struggles on small images and large scale factors.

  • DIP-FKP v.s. KernelGAN-FKP: DIP-FKP generally estimates kernels better and converges more stably for small images, while KernelGAN-FKP requires much less memory.DIP-FKP jointly estimates the kernel and HR image; KernelGAN-FKP optimizes only the kernel.
  • DIP-FKP v.s. KernelGAN-FKP: KernelGAN-FKP does not perform well for small images and large scale factors because it must re-downscale the LR image.This limitation contrasts with DIP-FKP's stronger behavior on small images.

7. Conclusion

FKP models kernel distributions with an invertible flow prior and integrates into existing blind SR methods. Experiments show improved kernel estimation and state-of-the-art blind SR results.

  • FKP learns an invertible mapping between complex kernel distributions and tractable latent-variable distributions using normalizing-flow blocks.
  • FKP freezes its trained parameters and optimizes latent variables, guaranteeing reasonable kernels during initialization and optimization.
  • FKP replaces kernel-modeling modules in methods such as Double-DIP and KernelGAN, improving kernel estimation accuracy and producing state-of-the-art blind SR results.
Loading 2103.15977v1…