PaperlayerView exact revision on arXiv

Source-linked AI summary

Invertible generative models for inverse problems: mitigating representation error and dataset bias

Muhammad Asim, Mara Daniels, Oscar Leong, Ali Ahmed, Paul Hand

arXiv:1905.11672v5cs.CV

TL;DR

GAN priors can be powerful for imaging inverse problems but may have representation error from architecture, mode collapse, and dataset bias. This paper uses pretrained invertible neural networks as likelihood-based priors, optimizing measurement fit in latent space with regularization or initialization. The priors outperform or match competing methods across several tasks and retain performance on out-of-distribution images, while the paper also provides linear-model recovery bounds.

  • Problem

    GAN priors may be unable to represent particular in-distribution or out-of-distribution images because of architectural choices, mode collapse, and training-set bias.

  • Method

    The paper uses pretrained invertible generators as priors and optimizes latent-space data fit while promoting high-likelihood representations through penalties or initialization.

  • Results

    Invertible priors improve denoising over BM3D and can outperform GANs, sparsity priors, and the Deep Decoder in compressive sensing, including for out-of-distribution images.

  • Takeaways & Limitations

    Zero representation error lets invertible priors mitigate dataset bias while supporting recovery across multiple imaging inverse problems.

Abstract

from arXiv · show

Trained generative models have shown remarkable performance as priors for inverse problems in imaging -- for example, Generative Adversarial Network priors permit recovery of test images from 5-10x fewer measurements than sparsity priors. Unfortunately, these models may be unable to represent any particular image because of architectural choices, mode collapse, and bias in the training dataset. In this paper, we demonstrate that invertible neural networks, which have zero representation error by design, can be effective natural signal priors at inverse problems such as denoising, compressive sensing, and inpainting. Given a trained generative model, we study the empirical risk formulation of the desired inverse problem under a regularization that promotes high likelihood images, either directly by penalization or algorithmically by initialization. For compressive sensing, invertible priors can yield higher accuracy than sparsity priors across almost all undersampling ratios, and due to their lack of representation error, invertible priors can yield better reconstructions than GAN priors for images that have rare features of variation within the biased training set, including out-of-distribution natural images. We additionally compare performance for compressive sensing to unlearned methods, such as the deep decoder, and we establish theoretical bounds on expected recovery error in the case of a linear invertible model.

1. Introduction

The paper studies invertible neural networks as image priors that avoid the representation error and dataset bias affecting GAN priors. Across denoising, compressive sensing, and theoretical recovery analysis, the proposed priors achieve strong reconstruction performance, including for out-of-distribution images.

  • Motivation: GAN priors can outperform sparsity methods with 5-10x fewer compressive-sensing measurements, but may fail to represent desired images because of architecture, mode collapse, and training bias.These representation errors affect both in-distribution and out-of-distribution images.
  • Proposed prior: Invertible neural networks are one-to-one and onto by design, giving zero representation error and permitting recovery of any image, including images far outside the training distribution.Their equal-dimensional latent and signal spaces support exact inversion and likelihood evaluation.
  • Approach: The method uses pretrained invertible generators as priors for multiple inverse problems without using problem-specific knowledge during training.Recovery seeks a high-likelihood image consistent with the measurements, using latent-space regularization or initialization.
  • Results: Invertible priors can produce sharper denoising results with higher PSNRs than BM3D.
  • Results: For compressive sensing, invertible priors can outperform GANs, sparsity priors, and the Deep Decoder across wide undersampling ranges, including on out-of-distribution images.On out-of-distribution images, their performance decays gracefully and can exceed low-dimensional GANs; with sufficiently many measurements, it can be higher or comparable to the Deep Decoder.
  • Theory: The paper proves upper and lower bounds on expected squared recovery error for linear invertible generators in terms of the smallest n −m singular values.The result concerns m linear measurements of an n-dimensional signal.

2. Method

The method uses a pretrained Glow-style invertible generator to map equal-dimensional latent and image spaces, then solves inverse problems by optimizing measurement fit in latent space. Likelihood is encouraged through latent regularization or initialization, while affine coupling layers preserve invertibility.

  • Model: The method assumes a pretrained invertible neural network G:R^n→R^n with x=G(z) and z=G^-1(x).The experiments use the Glow architecture.
  • Inverse formulation: The inverse-problem formulation seeks an image consistent with noisy linear measurements while favoring high likelihood under the invertible generator.The generator supplies likelihood estimates for all images.
  • Optimization: The implementation optimizes squared data fit over latent space, using likelihood penalties for denoising and algorithmic regularization for compressive sensing and inpainting.The latent initialization is z0=0 unless otherwise stated.
  • Optimization: The latent likelihood proxy treats images with small-norm latent representations as more natural among those consistent with undersampled measurements.This converts likelihood selection over an affine measurement set into a latent-space optimization problem.
  • Caveat: The likelihood proxy is imperfect because some high-likelihood latent codes can correspond to low-likelihood images.The paper states that such images have low total probability and are inconsistent with enough measurements.
  • Architecture: Glow uses affine coupling layers that transform half the input using learned scale and translation parameters computed from the other half, while remaining invertible.Pixelwise reshuffling, implemented with learned 1x1 convolutions in Glow, lets each input component affect each output component.

3. Applications

Across denoising, compressive sensing, and inpainting, Glow invertible priors provide strong reconstructions while avoiding GAN representation-error limitations. Their performance depends on likelihood regularization or initialization choices.

  • Denoising: Glow can outperform BM3D in denoising while producing sharper reconstructions.An appropriately regularized Glow prior exceeds BM3D by almost 2 dB; Glow recoveries are sharper at σ = 0.1.
  • Denoising: Glow outperforms DCGAN in denoising because DCGAN suffers from representation error.Glow performance rises and then falls as γ increases, reflecting a trade-off between fitting noise and enforcing data fit.
  • Compressive Sensing: Zero initialization can make likelihood penalization unnecessary for Glow in compressive sensing and inpainting.Other initializations may benefit from direct likelihood penalization, while zero-initialized Glow achieves high inpainting PSNRs with γ = 0.
  • Compressive Sensing: Across a wide range of undersampling ratios, Glow outperforms learned and unlearned alternatives on in-distribution compressive sensing images.The comparison includes GANs, IAGAN, Deep Decoder, and Lasso-DCT priors.
  • Compressive Sensing: Glow retains strong compressive-sensing performance under extreme undersampling and degrades gracefully on out-of-distribution images.GAN architectures saturate from representation error, whereas Glow recovers features underrepresented in CelebA, including darker skin tones and accessories.

4. Theory

The theory analyzes maximum-likelihood recovery for a linear invertible generative model under Gaussian measurements. Its bounds relate recovery error to the model’s singular values and compare invertible recovery with best low-dimensional approximations.

  • Linear invertible model: Theorem 1 bounds absolute expected squared recovery error using the singular values of an invertible linear generator.The result assumes Gaussian-generated signals, Gaussian measurement matrices, and 4 ≤ m < n.
  • Recovery bounds: The lower bound matches the error of the best m-dimensional signal model based on the top m variance directions.Thus, m Gaussian measurements incur at least the error associated with retaining only the highest-variance directions.
  • Recovery bounds: The upper bound is within a factor of m of the error from the best m − 2 dimensional model.Both bounds are expressed through the singular-value spectrum of the generator.
  • Recovery behavior: If σ_i = o(i^−1/2), expected recovery error decreases to a small value as m approaches n.This contrasts with low-dimensional GAN theory, where error saturates at the GAN’s representation error.
  • Proof: The theorem is proved in the Supplemental Materials.

5. Discussion

The paper argues that invertible priors reduce representation error across inverse problems, while highlighting computational and theoretical trade-offs that remain unresolved.

  • Empirical findings: Invertible priors can outperform BM3D, Deep Decoder, low-dimensional GANs, and Lasso across several inverse-problem settings.The paper reports denoising, compressive-sensing, and inpainting results, plus graceful performance drops on out-of-distribution images.
  • Empirical findings: For out-of-distribution FFHQ compressive sensing at m = 7,500 (≈20%) measurements, Glow and Deep Decoder recoveries are comparable by PSNR.The comparison includes Glow, PGGAN, and an overparameterized Deep Decoder prior.
  • Scope and novelty: A pretrained invertible generator can be reused across inverse problems without training a separate model for each problem.This distinguishes the paper from work that trains a separate invertible model for every forward process.
  • Representation and likelihood: Invertible models avoid representation error because their generators are onto, but every image remains in range, so likelihood-based regularization must distinguish desirable solutions.Their fully dimensional representation does not explicitly encode a low-dimensional natural-image manifold.
  • Theory: The linear theory bounds expected recovery error using the singular values of the invertible generator, while acknowledging that the bound’s multiplicative m term and summation index may be nonoptimal.The analysis compares the resulting bound with the best m-dimensional manifold approximation.
  • Open problems: Solving the 128×128 color-image optimization took about 11 minutes on an NVIDIA 2080 Ti, motivating architectures that balance representation error, training cost, and inversion cost.The discussion also calls for methods between low-dimensional and fully invertible priors that could be cheaper to train and use.

6.1. Proofs

The proofs derive expected recovery-error bounds for a linear invertible generator by expressing Gaussian image samples in latent coordinates and analyzing random measurement projections.

  • Setup: The proof writes x0 as Gz0 with z0 drawn from a standard Gaussian, reducing image-space error analysis to latent-space calculations.This representation follows from the assumed Gaussian signal distribution pG = N(0, GGT).
  • Estimator: The latent maximum-likelihood estimator is characterized explicitly because AG has full rank with probability 1.The estimator is then related to an orthogonal projection onto the range of GTAT.
  • Reduction: Rotational and unitary invariance allow the analysis to reduce without loss of generality to a diagonal generator G = diag(σ1, . . . , σn).The proof uses the singular value decomposition and Gaussian rotational invariance.
  • Bounds: The lower-bound argument uses the rank of the projection and the Eckart-Young theorem to compare random recovery with optimal low-rank approximation.The expected lower bound is obtained by averaging the projection-error inequality over Gaussian measurements.
  • Bounds: The upper bound applies a randomized low-rank approximation theorem to the sampled matrix GAT.The stated theorem assumes target rank k ≥ 2, oversampling m − k ≥ 2, and 4 ≤ m ≤ n after substitution.
  • Supporting lemmas: Two supplementary lemmas provide the Gaussian norm identity and the projector commutativity property needed in the proof.The first identifies the expected squared norm of Mz with the squared Frobenius norm, while the second handles unitary transformations of projections.

6.2. Models

The experiments use Glow, DCGAN, PGGAN, and Deep Decoder models at 64px and 128px resolutions, with test sets covering both CelebA-HQ-like and FFHQ out-of-distribution images.

  • Glow: Glow models use invertible flow steps composed of activation normalization, invertible 1 × 1 convolutions, and affine coupling layers.The number of flow steps and splitting operations varies by inverse problem and image resolution.
  • Glow: At 128px, Glow uses a modified implementation with ϵ = 0.0005 in Actnorm scale parameters to mitigate inversion instability.The modification prevents division-by-zero errors, and forward activations are also clipped.
  • GAN baselines: The learned baselines are a 64px DCGAN and a 128px PGGAN, both using five upsampling layers with differing upsampling implementations.DCGAN uses transpose convolutions, whereas PGGAN uses nearest-neighbor upsampling followed by convolution.
  • Unlearned baseline: The overparameterized Deep Decoder provides an unlearned compressive-sensing baseline, using five or six upsampling layers and 250 or 700 channels.The 64px and 128px configurations use k = 250 and k = 700 activation channels, respectively.
  • Parameterization: The model-parameter summary defines representation size as the number of optimizable image-representation parameters and model size as all parameters combined.It also reports the overparametrization ratio as a model-specific parameter comparison.
  • Evaluation data: Experiments use validation images from CelebA-HQ for in-distribution testing and FFHQ images for out-of-distribution testing, with no test image seen during training.Samples from the trained models and both test sets are shown in the accompanying figures.

6.3. Denoising: Additional Experiments

Additional denoising experiments compare Glow, DCGAN, and BM3D across noise levels and regularization choices. Glow consistently outperforms DCGAN, and suitable regularization can match or surpass BM3D.

  • Glow outperforms DCGAN uniformly across all regularization values because DCGAN has representation error.
  • At high noise σ = 0.2, appropriately regularized Glow can outperform BM3D, while offering comparable performance at lower noise levels.
  • For DCGAN, increased regularization harms performance, unlike Glow, whose performance improves and then declines as γ increases.
  • At σ = 0.2, Glow performs best with γ = 0.05; γ = 0.001 overfits and γ = 1 underfits.

6.4. Compressive Sensing: Additional Experiments

Additional compressive-sensing experiments evaluate Glow against alternative priors across measurement levels, image distributions, and initialization strategies. Glow performs strongly on out-of-distribution natural images but can become unstable for very low-likelihood representations.

  • Glow performs best with zero initialization and no explicit latent-likelihood penalization in 64px CelebA recovery using m = 5000 (≈50%) measurements.
  • Invertible priors show graceful performance decay on out-of-distribution images and can outperform GANs across a wide range of undersampling ratios.
  • For sufficiently many measurements, invertible priors can achieve higher or comparable PSNRs than the Deep Decoder.
  • Glow is effective for semantically unrelated natural images, assigns them relatively high likelihood compared with noisy images, and outperforms LASSO.
  • On semantically unrelated images with very low likelihood, Glow can become unstable because optimization enters representations where network outputs diverge.

6.5. Inpainting

Inpainting experiments compare Glow and DCGAN on in-distribution and out-of-distribution images. Glow produces sharper, more coherent reconstructions, but performance deteriorates on natural images increasingly unlike CelebA faces.

  • Glow produces noticeably higher-quality inpainted reconstructions than DCGAN on CelebA test images.
  • For out-of-distribution images semantically similar to CelebA faces, Glow achieves reasonable reconstructions while DCGAN suffers from representation error and data bias.
  • Glow inpainting performance deteriorates on houses, doors, and butterfly wings that deviate further from the training images.

6.6. Discussion

The discussion examines how latent-space perturbations relate to image-space changes and how Glow’s Jacobian behaves. Natural image directions are less sensitive than random perturbations, informing optimization behavior.

  • Random image-space perturbations induce larger latent-space changes than comparable natural perturbations.
  • Figure 28 measures average image-space change over 100 test images as latent perturbation size varies, with shaded 95% confidence intervals.
  • The Glow model’s Jacobian singular-value decay is examined at random points to further characterize its local behavior.
Loading 1905.11672v5…