PaperlayerView exact revision on arXiv

Source-linked AI summary

Amortised MAP Inference for Image Super-resolution

Casper Kaae Sønderby, Jose Caballero, Lucas Theis, Wenzhe Shi, Ferenc Huszár

arXiv:1610.04490v3cs.CVcs.LGstat.ML

TL;DR

Image super-resolution is underdetermined, so pixel-wise objectives can produce implausible reconstructions despite matching the low-resolution input. This paper amortises MAP inference with affine-consistent CNNs, reducing training to a cross-entropy objective and solving it with GAN, denoising, or density-model methods; GAN-based inference performs best on real images and connects to variational inference.

  • Problem

    Super-resolution has many plausible high-resolution explanations for one low-resolution input, while MSE and related losses can select low-probability solutions.

  • Method

    The paper projects CNN outputs onto the affine subspace of valid solutions and minimizes cross-entropy using GAN, denoiser-guided, or maximum-likelihood image-prior methods.

  • Results

    GAN-based AffGAN produces the most visually appealing results in high dimensions and achieves cross-entropy close to the MAP solution in the toy experiment.

  • Takeaways & Limitations

    Affine-projected amortised MAP inference preserves low-resolution consistency while retaining the computational benefits of neural-network inference.

  • Takeaways & Limitations

    MAP estimates depend on the chosen representation and may not be uniquely or specially meaningful in 24-bit RGB pixel space.

Abstract

from arXiv · show

Image super-resolution (SR) is an underdetermined inverse problem, where a large number of plausible high-resolution images can explain the same downsampled image. Most current single image SR methods use empirical risk minimisation, often with a pixel-wise mean squared error (MSE) loss. However, the outputs from such methods tend to be blurry, over-smoothed and generally appear implausible. A more desirable approach would employ Maximum a Posteriori (MAP) inference, preferring solutions that always have a high probability under the image prior, and thus appear more plausible. Direct MAP estimation for SR is non-trivial, as it requires us to build a model for the image prior from samples. Furthermore, MAP inference is often performed via optimisation-based iterative algorithms which don't compare well with the efficiency of neural-network-based alternatives. Here we introduce new methods for amortised MAP inference whereby we calculate the MAP estimate directly using a convolutional neural network. We first introduce a novel neural network architecture that performs a projection to the affine subspace of valid SR solutions ensuring that the high resolution output of the network is always consistent with the low resolution input. We show that, using this architecture, the amortised MAP inference problem reduces to minimising the cross-entropy between two distributions, similar to training generative models. We propose three methods to solve this optimisation problem: (1) Generative Adversarial Networks (GAN) (2) denoiser-guided SR which backpropagates gradient-estimates from denoising to train the network, and (3) a baseline method using a maximum-likelihood-trained image prior. Our experiments show that the GAN based approach performs best on real image data. Lastly, we establish a connection between GANs and amortised variational inference as in e.g. variational autoencoders.

1 INTRODUCTION

The paper frames super-resolution as an underdetermined inverse problem where MSE and MAE can produce implausible solutions, and proposes amortised MAP inference using affine-consistent CNNs. Three training approaches are introduced, with GAN-based inference performing best on real images.

  • Super-resolution estimates a high-resolution image from a low-resolution input, but many plausible high-resolution images may explain the same observation.
  • The AffGAN and AffDG models achieve cross-entropy values close to the MAP solution, while MSE and MAE perform worse; affine projections outperform soft constraints.
  • MSE and MAE select posterior means or medians, which can lie in low-probability regions, whereas MAP inference selects the posterior mode.
  • The proposed affine projection constrains CNN outputs to the affine subspace of high-resolution solutions consistent with the low-resolution input.
  • With this architecture, training reduces to minimizing cross-entropy between the model-output distribution and the high-resolution image distribution without matched input-output pairs.

2 RELATED WORK

The related-work discussion situates the paper’s GAN variant within divergence-minimization theory and distinguishes its instance-noise contribution from other GAN-stability methods.

  • GANs have been connected to Shannon-Jensen divergence, KL divergence, and more general f-divergence minimization.
  • The paper introduces instance noise as a GAN-stability technique and compares its theoretical motivation with one-sided label smoothing.

3 THEORY

The paper reformulates amortised SR MAP inference as cross-entropy minimisation while enforcing consistency with the observed low-resolution input through affine projection. It proposes GAN-, denoiser-, and density-model-guided optimisation methods.

  • MAP inference: Amortised inference trains an SR function to map each low-resolution observation directly to its MAP high-resolution estimate.The objective is the average log-posterior over low-resolution observations.
  • Likelihood and constraints: Because downsampling is linear, valid high-resolution solutions form an affine subspace defined by x = Ay.The likelihood is degenerate, so SR becomes a constrained optimisation problem.
  • Likelihood and constraints: The proposed affine projection guarantees Afθ(x) = x, ensuring every network output is consistent with its low-resolution input.The projection uses convolution and deconvolution layers; the deconvolution is the Moore-Penrose pseudoinverse rather than simply the transpose.
  • Cross-entropy formulation: With affine projection, the objective becomes minimising cross-entropy between the reconstructed-image distribution qθ and the high-resolution image distribution pY.This removes the need for matched input-output pairs and makes the problem resemble unsupervised learning or generative modelling.
  • Optimisation methods: AffGAN uses the projected SR function as a GAN generator, while AffDG backpropagates gradient estimates obtained from a trained Bayes-optimal denoiser.AffGAN minimises a KL-based objective; denoising estimates the image-prior score needed for gradient optimisation.
  • Optimisation methods: Instance noise addresses GAN instability by smoothing concentrated model and natural-image distributions before minimising their divergence.The noise level can be annealed during training.
  • Optimisation methods: AffLL fits a maximum-likelihood density model to natural images and uses its cross-entropy to approximate the amortised MAP objective.The baseline uses a differentiable MCGSM likelihood in a deep generative model similar to PixelCNN.

4 EXPERIMENTS

The experiments evaluate whether the proposed methods minimise cross-entropy, preserve CNN performance under affine projection, and improve perceptual SR quality. They use a tractable toy distribution and 4× downsampling on textures, faces, and natural images.

  • Experimental questions: The experiments target cross-entropy minimisation, the effect of affine projection on CNN performance, and perceptual superiority of the proposed SR methods.These questions are evaluated across Sections 5.1–5.5.
  • Datasets and settings: Exact MAP inference is computationally tractable for the initial two-dimensional noisy Swiss-roll experiment, where the low-resolution value averages two high-resolution pixels.This toy setting is used to illustrate algorithm behaviour.
  • Datasets and settings: Natural-image experiments use 4× downsampling on grass textures, Celeb-A faces, and ImageNet images.The texture task is chosen because SR with MSE or MAE is known to be difficult for random textures.

5 RESULTS AND DISCUSSION

Experiments show that affine projections enforce input-output consistency and support sharper, more plausible super-resolution than MSE-based alternatives, while GAN outputs retain important limitations.

  • 2D MAP inference: Swiss-roll: AffGAN and AffDG solutions largely fit the dominant prior mode, whereas MSE and MAE outputs generally fall in low-density regions.The affine-projected models achieve cross-entropy values relatively close to the optimal MAP solution.
  • 2D MAP inference: Swiss-roll: Affine projections exactly satisfy input consistency in the toy experiment, while soft-constrained models only approximately satisfy it.The affine projection models have zero MSE between the network input and downsampled output.
  • Grass textures: AffGAN produces the sharpest grass reconstructions and plausible texture statistics, while AffDG and AffLL remain very blurry.The reconstruction is not pixel-perfect but retains statistical properties recognizable as grass texture.
  • CelebA faces: AffGAN and SoftGAN produce sharper CelebA faces than blurry MSE outputs, with AffGAN slightly sharper but exhibiting slightly more high-frequency noise.The MSE-trained models achieve the best PSNR and SSIM scores, while affine projections provide better input-output consistency than soft constraints.
  • Natural images: AffGAN produces generally sharp and plausible ImageNet outputs, but they remain distinguishable from true images and can resolve ambiguous content incorrectly.A snake is super-resolved into water, which is wrong but plausible given the low-resolution input and image prior.
  • Criticism and future directions: MAP inference is representation-dependent, motivating posterior sampling as a potentially preferable route to plausible reconstructions.The paper connects stochastic AffGAN to amortised variational inference but leaves empirical studies for future work.

6 CONCLUSION

The paper develops approximate MAP inference methods for SR using affine projection and GAN-, denoising-, or density-model-based approaches. AffGAN produces the most visually appealing high-dimensional results, while future work extends it toward amortised variational inference.

  • The paper combines affine projection with GAN, denoising, and density-model methods for amortised MAP inference in SR.
  • AffGAN produced the most visually appealing results in high-dimensional experiments.
  • The authors plan a stochastic AffGAN extension interpretable as amortised variational inference.

A GENERATIVE ADVERSARIAL NETWORKS FOR MINIMISING KL-DIVERGENCE

This section derives the GAN discriminator and generator optimization relationship used to connect GAN training with KL-divergence minimization. The derivation assumes the discriminator can closely approximate a unique Bayes-optimal discriminator.

  • For a fixed generator, the discriminator maximizes expected log-likelihood over real and generated samples.
  • The Bayes-optimal discriminator follows from maximizing a log(x) + b log(1 − x) objective under equal class priors.
  • The analysis assumes the Bayes-optimal discriminator is unique and closely approximated by the neural discriminator.
  • The combined discriminator–generator optimization problem is modified to support the proposed GAN training rule.
  • The generator terms in the derivation are shown to equal the corresponding terms from KL[qG∥pY] after substituting the Bayes-optimal discriminator.

B AFFINE PROJECTION

The affine projection enforces consistency between SR outputs and their LR inputs through convolutional downsampling and a numerically optimized pseudoinverse implementation. Its gradients are obtained by the chain rule and act as high-pass filtered generator gradients.

  • The downsampling projection A is implemented as a fixed Gaussian strided convolution, while A+ is a numerically optimized transposed convolution.
  • The stochastic objective estimates spectral norms of A − ABA and B − BAB, enabling optimization of the projection through stochastic gradient descent.
  • Gradients for affine-projected SR models are derived using the chain rule.
  • The resulting projected gradient is essentially a high-pass filtered version of the gradient of gθ(x).

C INSTANCE NOISE

Instance noise addresses GAN instability by broadening distribution support, unlike label smoothing, which leaves many near-optimal discriminators. The resulting noisy-distribution divergence is better behaved and improved AffGAN convergence in the experiments.

  • GAN training is described as alternating discriminator training, likelihood-ratio estimation, and a generator gradient step.
  • In low-dimensional well-conditioned settings, the procedure approximates gradient descent on KL divergence, but practical training can be highly unstable.
  • Non-overlapping concentrated supports can make KL divergence infinite, saturate Jensen–Shannon divergence, and produce diverse near-optimal discriminator gradients.
  • One-sided label smoothing makes discrimination harder but still leaves a large set of near-optimal classifiers when the Bayes-optimal classifier is non-unique.
  • Instance noise broadens both distributions, removes perfect separability, and makes the Bayes-optimal discriminator unique and the likelihood ratio better behaved.
  • Instance noise improved AffGAN convergence, while the authors had not tested it in generative modelling applications.

D EXPERIMENTAL DETAILS

The experiments use affine projections and several guided-training procedures across toy and real-image settings, with implementation choices specified for each dataset and model.

  • The generator and discriminator parameters for GAN models were updated using Eqn. (10), while soft-constraint models added an MAE loss term.
  • Denoiser-guided models first pre-trained a denoising autoencoder, then trained the generator using gradients estimated by the denoiser.
  • The denoiser noise level was annealed downward so training progressed from broad approximate gradients toward more precise gradients near the data manifold.
  • Density-guided models pre-trained a density model by maximizing tractable log-likelihood, then minimized generated-sample negative log-likelihood.
  • The experiments included a 2D Swiss-Roll toy distribution, grass textures, CelebA, and ImageNET, with dataset-specific convolutional architectures.
  • Image experiments used a Gaussian smoothing convolution with stride 4 for downsampling and a learned convolutional operation for 4× upsampling.

E ADDITIONAL RESULTS FOR DENOISER AND DENSITY GUIDED SUPER-RESOLUTION

Additional results examine denoiser- and density-guided super-resolution on grass textures and show AffGAN outputs for 4× ImageNET super-resolution.

  • The AffDG and AffLL grass-texture models converged during training, but their generated images remained very blurry.
  • Both AffDG and AffLL experienced training divergence, motivating schedules that moved from high-noise or low-likelihood models toward more precise models.
  • AffDG training displayed step-like behaviour because denoiser parameters were replaced with versions trained at continuously lower noise levels.
  • Figure 8 presents 4× ImageNET super-resolution from 32 × 32 inputs to 128 × 128 outputs, comparing AffGAN outputs with true high-resolution images.

F AMORTISED VARIATIONAL INFERENCE USING AFFGAN

The stochastic AffGAN extension is connected to amortised variational inference by using independent noise alongside the low-resolution input. The resulting criterion requires only samples from the low- and high-resolution data distributions.

  • The stochastic AffGAN generator takes both a low-resolution image x and independent noise variables z.
  • The affine projection ensures that generated low- and high-resolution variables remain consistent under the joint model.
  • The paper concludes that AffGAN approximately minimizes an amortised variational inference criterion.
  • The criterion can be optimized using only samples from the high-resolution and low-resolution data distributions.
Loading 1610.04490v3…