PaperlayerView exact revision on arXiv

Source-linked AI summary

SRFlow: Learning the Super-Resolution Space with Normalizing Flow

Andreas Lugmayr, Martin Danelljan, Luc Van Gool, Radu Timofte

arXiv:2006.14200v2cs.CVeess.IV

TL;DR

Super-resolution is ill-posed because one low-resolution image can correspond to many high-resolution images, whereas common methods usually learn a single output. SRFlow uses a conditional normalizing flow trained with negative log-likelihood to model this distribution, and experiments report better perceptual and reconstruction quality than GAN-based approaches while enabling diverse samples and image manipulation.

  • Problem

    Super-resolution methods commonly learn deterministic mappings or single outputs, despite the existence of multiple plausible high-resolution images for each low-resolution input.

  • Method

    SRFlow uses a conditional normalizing flow trained with a single negative-log-likelihood loss to learn the conditional distribution of plausible high-resolution images.

  • Results

    SRFlow outperforms state-of-the-art GAN-based methods on perceptual and reconstruction-based metrics in face and general super-resolution experiments.

  • Takeaways & Limitations

    The learned high-resolution distribution supports diverse super-resolution samples and image manipulation, including editing and content transfer.

Abstract

from arXiv · show

Super-resolution is an ill-posed problem, since it allows for multiple predictions for a given low-resolution image. This fundamental fact is largely ignored by state-of-the-art deep learning based approaches. These methods instead train a deterministic mapping using combinations of reconstruction and adversarial losses. In this work, we therefore propose SRFlow: a normalizing flow based super-resolution method capable of learning the conditional distribution of the output given the low-resolution input. Our model is trained in a principled manner using a single loss, namely the negative log-likelihood. SRFlow therefore directly accounts for the ill-posed nature of the problem, and learns to predict diverse photo-realistic high-resolution images. Moreover, we utilize the strong image posterior learned by SRFlow to design flexible image manipulation techniques, capable of enhancing super-resolved images by, e.g., transferring content from other images. We perform extensive experiments on faces, as well as on super-resolution in general. SRFlow outperforms state-of-the-art GAN-based approaches in terms of both PSNR and perceptual quality metrics, while allowing for diversity through the exploration of the space of super-resolved solutions.

1 Introduction

SRFlow addresses the ill-posed nature of super-resolution by learning a conditional distribution of plausible high-resolution images rather than a single deterministic output. Its flow-based formulation supports diverse photo-realistic samples and image manipulation while achieving strong reconstruction and perceptual quality.

  • For one low-resolution image, infinitely many compatible high-resolution predictions can exist, making super-resolution fundamentally ill-posed.
  • Reconstruction-loss methods often produce blurry averages, while adversarial and perceptual methods generate sharper but still single outputs.
  • SRFlow learns the conditional distribution of plausible high-resolution images using a conditional normalizing flow and exact negative-log-likelihood training.
  • The learned distribution enables sampling multiple diverse, photo-realistic super-resolved images consistent with the input low-resolution image.
  • Compared with GAN-based approaches, SRFlow avoids mode collapse, uses a single training loss, and supports invertible image editing.
  • Comprehensive face and general-image experiments report superior perceptual and reconstruction-based metrics to state-of-the-art GAN-based methods.

2 Related Work

Prior super-resolution methods generally predict one sharp-looking reconstruction, while stochastic super-resolution seeks the broader distribution of plausible outputs. SRFlow extends conditional flow modeling to this setting and adds image manipulation and restoration applications.

  • Single image SR: Earlier deep super-resolution methods relied on L1 or L2 losses and failed to reproduce sharp natural-image details.
  • Single image SR: Conditional GAN approaches improved sharpness and perceptual quality but generally predicted only one example rather than all plausible reconstructions.
  • Conditional flows: Conditional flow models compute exact negative log-likelihood through the change-of-variable formula and can be trained by minimizing it directly.
  • Related conditional-flow work: A concurrent conditional-flow work addressed only 2× super-resolution and lacked comparisons with state-of-the-art GAN-based methods.
  • SRFlow's scope: SRFlow differs by targeting competitive super-resolution, flow-based manipulation, new training and architectural considerations, and restoration tasks unseen during training.

3 Proposed Method: SRFlow

SRFlow models the conditional distribution of plausible high-resolution images with an invertible, multi-level normalizing flow conditioned on the low-resolution image. Exact likelihood training enables sampling and image manipulation while preserving efficient inversion and tractable density computation.

  • Conditional Normalizing Flows: SRFlow learns rich conditional HR-image distributions by mapping HR images to latent variables with an invertible flow conditioned on an encoded LR image.The inverse mapping reconstructs HR images from sampled latent variables under a simple latent distribution such as a Gaussian.
  • Conditional Normalizing Flows: The model trains by minimizing the negative log-likelihood computed with the change-of-variables formula and Jacobian determinants.The flow is decomposed into invertible layers so the Jacobian log-determinant remains tractable.
  • Inference: At inference, SRFlow samples latent variables to generate HR images, with Fig. 3 illustrating random 8× samples at temperature τ = 0.8.The latent representation supports exploration of alternative super-resolved outputs.
  • Architecture: Each flow-step combines Actnorm, an invertible 1 × 1 convolution, an Affine Injector, and a Conditional Affine Coupling layer.The Affine Injector uses the LR encoding to predict element-wise scaling and bias across all channels and spatial locations.
  • Architecture: The architecture uses multiple resolution levels, squeeze operations, channel splitting, and spatially distributed latent variables to model image variations at different scales.After squeezing, early flow steps omit conditional layers to learn a linear invertible interpolation intended to reduce checkerboard artifacts.
  • Training Details: Training uses batches of 16 LR-HR pairs and 160 × 160 HR patches, with Adam optimization after pre-training the LR encoder using an L1 loss.The full network is then trained with the negative log-likelihood objective.

4 Applications and Image Manipulations

SRFlow exploits a learned HR-image distribution and invertible latent representation for diverse sampling, style/content transfer, image manipulation, and restoration. Latent normalization adapts encoded images to the LR-conditioned distribution while preserving or restoring desired content.

  • Core capabilities: SRFlow models a conditional HR-image distribution and uses its invertible encoder-decoder to operate flexibly in latent and image space.Any HR image can be encoded into latent space and exactly reconstructed, enabling image-space and latent-space manipulation.
  • Diverse super-resolution: Sampling different latent variables produces diverse SR predictions, including variations in hair and facial attributes, while preserving LR consistency.The experiments use Gaussian latent variables with temperature τ = 0.8.
  • LR-Consistent Style Transfer: Style transfer encodes an existing HR source image and reuses its latent code to super-resolve another LR image with transferred characteristics.The operation can transfer facial characteristics, hair, and eye color, including on local regions.
  • Latent Space Normalization: Latent normalization maps arbitrary HR content toward statistics consistent with the given LR image, enabling controlled manipulation and stochastic diverse outputs.Normalization can be applied globally, locally, or spatially, depending on the latent collection being normalized.
  • Image Content Transfer: For content transfer, SRFlow locally normalizes only affected regions to harmonize inserted content while preserving non-manipulated image regions.Training with randomly misaligned HR-LR pairs can reduce emphasis on LR consistency and increase manipulation flexibility.
  • Image Restoration: For restoration, SRFlow encodes a degraded image, applies spatial and local latent normalization, and predicts a cleaner image from the LR-conditioned distribution.The method recovers substantial detail by transforming the degraded image toward a more likely clean image under the learned SR distribution.

5 Experiments

SRFlow is evaluated on face and general super-resolution, with comparisons against reconstruction- and GAN-based methods. It achieves stronger perceptual and fidelity results in reported experiments, while architectural ablations examine flow depth, width, and conditioning.

  • Experimental setup: Experiments cover 8× face super-resolution on CelebA and general super-resolution on DIV2K, using comparisons with current methods and ablations.Face evaluation uses 5,000 CelebA test images; general SR uses the DIV2K validation set.
  • Face Super-Resolution: SRFlow achieves more than twice as good LPIPS distance as RRDB for face SR, although RRDB has higher PSNR and SSIM.LPIPS is treated as the primary metric because it better correlates with human opinions.
  • Face Super-Resolution: Compared with GAN-based face SR methods, SRFlow achieves significantly better results across the reported reference metrics while preserving fidelity and perceptual quality.The authors attribute visual differences partly to hallucination artifacts in GAN-based approaches.
  • General Super-Resolution: For general SR, SRFlow achieves significantly better PSNR, LPIPS, and LR-PSNR than GAN-based methods, with favorable PIQUE and BRISQUE results.Visual comparisons report richer details than EDSR and RRDB and fewer discoloration and ringing artifacts than ESRGAN.
  • Ablation analysis: Reducing flow steps increases artifacts in complex structures, while increasing conditional-layer channels improves reconstruction consistency; the affine injector increases fidelity.These effects are reported in CelebA ablations and architectural analyses.

6 Conclusion

SRFlow learns a distribution of photo-realistic high-resolution images conditioned on the input low-resolution image. The paper also uses this learned distribution for diverse sampling and image manipulation, reporting improved results over state-of-the-art GAN-based approaches.

  • Conclusion: SRFlow explicitly models the ill-posed super-resolution problem by learning a conditional distribution of diverse photo-realistic high-resolution images.The distribution is conditioned on the input low-resolution image.
  • Conclusion: The learned image posterior supports image manipulation techniques, including guided super-resolution and content transfer.These applications exploit the strong high-resolution distribution learned by SRFlow.
  • Conclusion: Comprehensive experiments report improved results compared with state-of-the-art GAN-based approaches.

A Architecture Details

SRFlow uses a fully convolutional, invertible flow architecture conditioned on a low-resolution image encoding. Its affine injector directly conditions flow features while preserving invertibility, and Actnorm supplies per-dimension learned normalization.

  • Architecture: The architecture is fully convolutional and invertible, allowing training on smaller patches and application to full images at test time.Invertible flow layers also require a tractable Jacobian log-determinant.
  • Low-resolution conditioning: The flow is conditioned on a low-resolution encoding produced by an RRDB-based encoder with 320 output channels.The encoding concatenates activations from five equally spaced RRDB blocks.
  • Affine injector: The affine injector conditions all flow-feature dimensions on low-resolution features without harming invertibility.It applies an affine transformation to part of the feature map while bypassing the remainder.
  • Affine injector: Non-invertible networks extract the scale and bias, after resizing the low-resolution encoding to the corresponding flow level.Separate convolutional layers produce the scale and bias outputs.
  • Squeeze operation: The squeeze operation halves activation-map width and height by stacking neighboring pixels to preserve locality.
  • Normalization: Actnorm uses learned scaling and bias factors for each feature-map dimension and does not require batch-element synchronization.This supports distributed learning across multiple GPUs.

B Training Details

Training uses Adam with scheduled learning-rate reductions and a warm-up phase that freezes the low-resolution encoder. Gaussian noise is added to targets, while the section also presents visualization and quantitative-analysis material.

  • Optimization: Training starts with Adam at a learning rate of 5·10^-4, halved at 50%, 75%, 90%, and 95% of the iterations.
  • Optimization: The pretrained low-resolution encoder remains frozen during the first 50% of training, after which all SRFlow parameters are jointly optimized.
  • Target preprocessing: Gaussian noise with standard deviation σ = 4√3 is added to high-resolution targets to improve training and visual results.Unlike the cited prior approach, SRFlow does not use 5-bit quantization.
  • Visual analysis: The section includes visualizations of the squeeze operation and super-resolved images sampled at different temperatures.
  • Quantitative analysis: Additional quantitative analysis is provided for the approach.

C.1 Influence of the Sampling Temperature

Sampling temperature controls SRFlow’s balance between fidelity, perceptual quality, and diversity. Lower temperatures favor fidelity, while higher temperatures improve perceptual quality and sampling diversity.

  • Temperature effects: At τ = 0, sampling is deterministic and produces high PSNR and SSIM but blurry images with poor LPIPS perceptual quality.Reducing temperature below 1 increases image quality, but the zero-temperature setting removes stochastic diversity.
  • Temperature effects: Increasing τ improves perceptual quality in LPIPS and increases sampling diversity measured by pixel-wise variance.The figures analyze these effects on CelebA at 8× and DIV2K at 4× and 8×.
  • Evaluation: The analysis evaluates sampling temperature using PSNR, SSIM, LPIPS, and sample diversity across face and general super-resolution settings.RRDB and ESRGAN are included as reference methods in the plotted comparisons.

C.2 Perception–Distortion analysis

SRFlow exposes a controllable perception–distortion trade-off through sampling temperature, unlike reference methods that provide only fixed operating points. Its pretrained LR-Encoder offers comparable perceptual quality and slightly higher fidelity than random initialization.

  • Perception–distortion analysis: Varying τ produces alternative perception–distortion operating points, allowing SRFlow to control the trade-off during inference.RRDB and ESRGAN each provide only a single operating point in the comparison.
  • Perception–distortion analysis: At τ = 0, SRFlow achieves similar PSNR to the L1-trained RRDB baseline.This setting represents the high-fidelity end of SRFlow’s controllable trade-off.
  • Perception–distortion analysis: For τ ≥ 0.8, SRFlow provides similar or better LPIPS perceptual quality than ESRGAN while achieving superior PSNR fidelity.The comparison is reported against the ground truth in the LPIPS-versus-PSNR analysis.
  • LR-Encoder pretraining: Pretraining the LR-Encoder yields comparable perceptual quality and slightly higher fidelity than random initialization.The pretrained model and the non-pretrained model use different training durations in the comparison.

C.4 Oracle Analysis of the Sampling Space

SRFlow’s sampled solution space supports oracle selection and restoration beyond ordinary super-resolution. Selecting among multiple samples especially improves perceptual quality, while latent space normalization enables denoising despite training only on clean images.

  • Oracle sampling: Selecting the best of n random samples at τ = 0.8 particularly improves perceptual quality on the 5000-image CelebA test set.The analysis evaluates n = 1 through 10 and selects the best sample according to each performance metric.
  • Oracle sampling: SRFlow’s diverse super-resolved samples allow lower-quality predictions to be rejected and higher-quality solutions to be selected.A visual LPIPS-based best-of-n example is provided alongside the quantitative analysis.
  • Image restoration: Latent space normalization uses the original image for restoration, preserving details that direct super-resolution may remove while denoising.The restoration examples cover CelebA degradations and DIV2K denoising.
  • Image restoration: For white Gaussian noise with σ = 20, SRFlow produces promising denoising results despite being trained only to super-resolve clean images.Quantitative results are reported on CelebA and DIV2K against clean ground truth.

D Visual Results

The visual results show SRFlow’s comparisons with state-of-the-art methods, diverse samples, and image manipulation capabilities across face and general super-resolution. The examples cover multiple scales, degradation settings, and content-transfer scenarios.

  • D.1 State-of-the-Art for Face Super-Resolution: On CelebA face super-resolution, SRFlow provides superior perceptual quality and better fidelity than the compared GAN-based methods.Figure 20 includes bicubic and bilinear downsampling settings, with SRFlow trained and applied under the corresponding kernels.
  • D.2 State-of-the-Art General Super-Resolution: On DIV2K at 4× and 8×, SRFlow achieves perceptual quality similar to or better than ESRGAN in most cases without typical GAN hallucination artifacts.The comparisons are shown on the DIV2K validation sets.
  • D.3 Random Sampling: Random SR samples remain consistent with the LR input while varying in facial characteristics, textures, and other fine details.The examples span CelebA at 8× and 16× and DIV2K at 4× and 8×.
  • D.4 Image Content Transfer: Latent space normalization integrates transferred image content into both existing HR images and SR predictions.Additional content-transfer examples use a model trained with random 4px HR shifts for greater flexibility.
Loading 2006.14200v2…