PaperlayerView exact revision on arXiv

Source-linked AI summary

Intermediate Layer Optimization for Inverse Problems using Deep Generative Models

Giannis Daras, Joseph Dean, Ajil Jalal, Alexandros G. Dimakis

arXiv:2102.07364v1cs.LG

TL;DR

Inverting deep generators to explain measurements is difficult, motivating approximate methods for inverse problems. ILO progressively optimizes intermediate layers with constrained deviations, and the authors report strong performance across several inverse-problem settings.

  • Problem

    Finding a latent code that explains measurements can be difficult because inverting even a four-layer generator is NP-hard.

    Source: §1 · p. 1

  • Method

    ILO progressively optimizes intermediate generator layers, searching within small l1 balls around the previous layer’s manifold.

    Source: §1 · p. 1, §2.2 · p. 2

  • Results

    ILO largely outperforms alternative techniques in visual quality and true MSE across inpainting, super-resolution, denoising, and compressed sensing with circulant matrices.

    Source: §4.3 · p. 6

  • Takeaways & Limitations

    The experiments show how pretrained generators combined with differentiable forward operators can guide data-driven image reconstruction.

    Source: §5 · p. 9

    Tuning the radius for each layer is difficult, and these hyperparameters do not transfer across tasks.

    Source: §7.3 · p. 14

Abstract

from arXiv · show

We propose Intermediate Layer Optimization (ILO), a novel optimization algorithm for solving inverse problems with deep generative models. Instead of optimizing only over the initial latent code, we progressively change the input layer obtaining successively more expressive generators. To explore the higher dimensional spaces, our method searches for latent codes that lie within a small $l_1$ ball around the manifold induced by the previous layer. Our theoretical analysis shows that by keeping the radius of the ball relatively small, we can improve the established error bound for compressed sensing with deep generative models. We empirically show that our approach outperforms state-of-the-art methods introduced in StyleGAN-2 and PULSE for a wide range of inverse problems including inpainting, denoising, super-resolution and compressed sensing.

1. Introduction

The paper studies unsupervised image reconstruction with pretrained generators, where inversion can be difficult, especially for deeper models. It introduces ILO, which optimizes intermediate layers with small deviations and reports theoretical and empirical benefits across inverse problems.

1. Introduction

  • Inverting even a four-layer generator is NP-hard, motivating approximate methods for finding latent codes that explain measurements.The paper focuses on unsupervised reconstruction using pretrained generators.Source: §1 · p. 1

1.1. Our Contributions

  • ILO adaptively optimizes different generator layers, extending the generator’s range while constraining intermediate searches to small l1 balls around the previous layer’s manifold.The method is presented as extending PULSE beyond super-resolution to inverse problems with differentiable forward operators.Source: §1.1 · p. 1, §1 · p. 1
  • Restricting the radius of intermediate latent searches can improve the established CSGM error bound, while the paper establishes sample-complexity and error bounds for ILO.The paper relates its layer-wise deviations to sparse deviations in image space, extending that framework to the manifold of any generator layer.Source: §1.1 · pp. 1–2
  • Across inpainting, denoising, and super-resolution, the authors report that ILO significantly outperforms previous state-of-the-art techniques for inverse problems with deep generative models.They also demonstrate classifier-based measurements that guide generators toward human images resembling ImageNet classes, and open-source their code.Source: §1.1 · p. 2

2. Algorithm

ILO splits a pretrained generator into two parts and first optimizes the original latent code. It then searches nearby intermediate representations, projecting each solution back toward the generator’s range while seeking measurements that explain the image.

2.1. Setting

  • The pretrained generator is decomposed as G = G2 ◦ G1, mapping an initial latent code through an intermediate space to the image.The initial latent codes are sampled from a simple distribution, typically Gaussian and independent.Source: §2.1 · p. 2

2.2. Approach

  • ILO begins by optimizing the k-dimensional input code as in CSGM, then optimizes the input to G2 within an l1 ball centered on the previous intermediate representation.The bounded deviations expand the generator’s range while restricting departures intended to avoid overfitting to measurements.Source: §2.2 · p. 2
  • After each intermediate search, ILO finds an initial latent code whose G1 output is close to that solution, then maps it back to the intermediate space for another round.The method seeks the measurement-fitting latent vector in the extended range formed by dilating G1’s range with an l1 ball.Source: §2.2 · p. 2
  • The theoretical bounds concern the global optimum in the extended range, but projected gradient descent solves a non-convex problem and can become stuck in local optima.The authors leave open whether such local optimization can be shown to reach global minima under generator-weight assumptions.Source: §2.2 · p. 2

3. Theoretical Analysis

The theoretical analysis bounds reconstruction over an extended generator range formed by small l1 deviations from the first layer’s range. It relates measurement requirements and error to the radius of that expansion and also establishes a result for randomly signed partial circulant measurements.

3.1. Preliminaries

  • S-REC requires measurements to distinguish sufficiently different vectors in a set, a property used to prove small reconstruction error from observations Ax.The cited CSGM analysis establishes this property for the generator range under Gaussian measurements.Source: §3.1 · pp. 2–3

3.2. Intermediate Layer Optimization

  • With sufficiently many measurements, minimizing measurement error over the extended range yields a reconstruction close to the best reconstruction that range can produce.The range consists of vectors within an l1 ball of radius r2 from points in G1’s range; the theorem states that the measurements optimum is nearly as good as the true optimum.Source: §3.2 · p. 4

These many measurements result in an additive error term

  • For K scaling approximately as k, measurements scale linearly with k and logarithmically with p rather than linearly with p, at the cost of an additive error scaling with √p and a restricted search radius.The extended-range size and error bound depend on K; increasing K expands the search and reduces both terms in the bound, but measurements scale quadratically with K.Source: §3.2 · p. 4

3.3. Sketch of the proof

  • The proof bounds metric entropy for the extended range by covering the first generator’s range and the l1 ball, using Maurey’s method for logarithmic rather than linear dependence on dimension p.The chaining argument switches from volumetric epsilon-nets to Maurey’s method at a selected scale to reduce additive error.Source: §3.3 · p. 5

3.4. S-REC for partial circulant matrices

  • Randomly signed partial circulant matrices satisfy S-REC with a number of measurements scaling similarly to i.i.d. Gaussian measurements.For high-resolution sensing, their structure requires much less memory and enables faster matrix-vector products via FFT than Gaussian measurement matrices.Source: §3.4 · p. 5

4. Experiments

ILO adapts StyleGAN-2 inversion by progressively optimizing intermediate layers, with task-specific losses and regularization. Across four inverse-problem settings, it achieves state-of-the-art unsupervised performance and generally outperforms prior methods.

4.1. Algorithmic adaptations to StyleGAN

  • For StyleGAN-2, ILO runs in rounds over multiple generator splits, discarding the initial layer at each round.The generator is treated as a composition of 18 layers.Source: §4.1.1 · p. 5

4.1.1. OPTIMIZATION SETTING

  • ILO uses PGD to keep searches within an l1 ball around the current best solution, increasing the allowed deviation in higher-dimensional spaces.Ball radii are tuned separately as hyperparameters.Source: §4.1.1 · pp. 5–6

4.2. Loss functions and adaptation to general inverse problems

  • For denoising, ILO adds Gaussian noise to generated images before computing the loss, a technique called Stochastic Noise Addition (SNA).For inpainting with scarce measurements, it uses MSE alone; with sufficient measurements and for super-resolution, it uses weighted MSE and LPIPS.Source: §4.2 · p. 6

4.3. Results

  • ILO achieves state-of-the-art unsupervised performance across inpainting, super-resolution, denoising, and compressed sensing with circulant matrices.Figure 2 reports true MSE on Celeba-HQ and shows ILO outperforming prior methods except in the very noisy regime.Source: §4.3 · p. 6, Figure 2 · p. 7
  • ILO is 2× better in reconstruction error than other methods across 5%–100% observed pixels in inpainting.Experiments include randomized inpainting with observation probabilities as low as 1%.Source: §4.3 · p. 6
  • At σ = 25, ILO is 1.8× better than the best-performing CSGM baseline for denoising.BM3D outperforms the other methods in the very low-noise regime but deteriorates for harder settings.Source: §4.3 · p. 7
  • At 16× downscaling, ILO is 1.65× better than PULSE in reconstruction error; at 32 × 32, it is 1.4× better.These super-resolution comparisons use Celeba-HQ images and bicubic downsampling.Source: §4.3 · p. 7
  • With partial circulant measurements equal to 5% of dimension n, ILO achieves 2× lower reconstruction error than CSGM.The experiment uses partial circulant matrices with random signs.Source: §4.3 · p. 7

5. Conclusions and Future Work

ILO expands the generator’s range by optimizing intermediate layers and performs well across several inverse-problem tasks. The authors identify global convergence and application to domains such as medical imaging as open directions.

5. Conclusions and Future Work

  • ILO expands the range of pre-trained generators by optimizing different intermediate layers and achieves strong performance on several inverse-problem tasks.The authors identify establishing global convergence as an open theoretical problem.Source: §5 · p. 9
  • Applying the framework to domains such as medical imaging remains an open empirical direction, requiring suitable pre-trained generators such as high-resolution MRI models.The authors also describe classifier-guided generation of out-of-distribution samples as an exciting direction.Source: §5 · p. 9

6. Intended Use

The authors frame ILO and their StyleGAN2 classifier use as a proof of concept and art project, not for identifying or manipulating real subjects. They also flag dataset imbalance and possible reconstruction bias as issues under investigation.

6. Intended Use

  • ILO is presented as a proof of concept, while the StyleGAN2 and classifier implementation is intended purely as an art project.The authors state that the model is not suitable for face recognition, identifying real subjects, or manipulating real-subject images.Source: §6 · p. 9
  • The authors do not publicly release the classifier transformation code because of its potential for abuse.Source: §6 · p. 9
  • The StyleGAN training dataset is noted to contain an imbalance of white faces relative to faces of color, and possible reconstruction-method bias remains under investigation.The authors suggest a FairFace-trained model as a possibility, while describing this as ongoing research.Source: §6 · p. 9

Symbols

This appendix defines notation and develops metric-entropy and S-REC arguments for nested generator layers. It then applies those arguments to the main theorem and a partial-circulant measurement setting.

Symbols

  • The notation section defines covering and packing numbers, norm balls, Minkowski sums, Gaussian complexity, and the index set [M].Source: Symbols · p. 10

7.1.1. METRIC ENTROPY FOR THE l1 BALL

  • The l1-ball entropy discussion gives alternative covering-number bounds: Maurey’s method trades logarithmic dimension dependence for polynomial dependence on 1/epsilon, while the volumetric argument reverses that trade-off.The appendix also invokes Sudakov’s minoration as yielding an epsilon-net size bound up to constants.Source: §7.1.1 · pp. 10–11

7.1.2. S-REC

  • The S-REC analysis considers a composition of Lipschitz generators and random Gaussian measurements, using nested nets to control measurement distortion.The proof chooses net scales and covering bounds to obtain high-probability control over successive net differences.Source: §7.1.2 · pp. 11–13

7.1.3. PROOF OF MAIN THEOREM 

  • The main-theorem proof notes that setting gamma to 4/5 recovers the same scaling term as CSGM.Source: §7.1.3 · p. 13

7.1.4. PROOF OF LEMMA 1

  • For a partial-circulant matrix with random diagonal signs, the appendix establishes S-REC using a Johnson–Lindenstrauss guarantee and an operator-norm bound.The stated argument relies on the JL guarantee for a set of size 2^O(m) and on ||FD||op <= sqrt(n).Source: §7.1.4 · pp. 13–14

7.2. Code

The repository provides PyTorch code and utilities for running and evaluating ILO across several image-reconstruction tasks. It also includes tools for classifier-based out-of-distribution generation and experiment visualization.

7.2. Code

  • The repository provides PyTorch code based on open-source StyleGAN2 implementations and inspired by an open-source PULSE implementation.A TensorFlow implementation was described as in progress.Source: §7.2 · p. 14
  • Included utilities cover setup, image preprocessing, and examples for inpainting, denoising, super-resolution, and compressed sensing with circulant matrices.Preprocessing examples include random inpainting, masks, noise addition, and face alignment.Source: §7.2 · p. 14
  • The repository includes code for out-of-distribution generation across 1,000 ImageNet classes using a robust classifier.Source: §7.2 · p. 14
  • Code is available to evaluate ILO against previous methods on Celeba-HQ and to visualize performance and track experiments.The code is GPU/CPU compatible and includes GIF-generation utilities for optimization frames.Source: §7.2 · p. 14

7.3. Experimental details

The experiments use projected gradient descent with layer-specific optimization settings and task-dependent losses. Deep-layer projection enables optimization that otherwise is not possible, but increases optimization steps and requires difficult, task-specific radius tuning.

7.3. Experimental details

  • Each inverse problem takes less than a minute on one 1080Ti GPU, and all experiments can be reproduced in less than a day on one GPU.Source: §7.3 · p. 14
  • Adam starts at learning rate 0.1 per layer, with a linear ramp-up and cosine ramp-down during each layer optimization.The schedule follows Karras et al. (2020).Source: §7.3 · p. 14
  • For random inpainting, MSE and LPIPS are used above 20% observed pixels, while only MSE is used below that threshold.For black-box inpainting both losses are used; all tasks use geodesic loss with coefficient 0.01.Source: §7.3 · p. 14
  • Projected gradient descent projects latent codes to the unit sphere and deeper-layer variables into l1 balls around the previous-layer manifold.The projections cover previous solutions, latent codes, and noise vectors, with separately tuned radii for variables and layers.Source: §7.3 · p. 14
  • Deep-layer l1 projection enables optimization that is otherwise impossible, improving reconstruction at the cost of more optimization steps.Tuning radii is especially difficult, and the hyperparameters do not transfer across tasks.Source: §7.3 · p. 14
  • Figure 2 reports averages across five Celeba-HQ images of each image’s best score over five runs per point, with error bars across experiments for different images.The authors searched five layer-step schedules, for 25 runs per method and horizontal-axis point.Source: §7.3 · p. 14

7.4. Additional Experiments

Additional experiments report denoising comparisons, classifier-guided image morphing, and inpainting with a cat-trained StyleGAN-2. The denoising results favor ILO visually and in performance over adapted PULSE and BM3D.

7.4. Additional Experiments

  • For classifier-guided morphing, MSE and LPIPS are used in early ILO rounds, with the classification term added in deeper generator layers.The reported benefit is that interpolated intermediate frames show a face matching the phase before the classifier alters it toward the ImageNet class.Source: §7.4 · p. 15, Figure 5 · p. 16
  • ILO gives superior visual reconstructions and better denoising performance than adapted PULSE and BM3D.Figure 6 presents denoising with known Gaussian noise at σ = 25; its MSE images show reconstructions obtained by inverting the noisy image.Source: §7.4 · p. 15, Figure 6 · p. 16
  • The authors also demonstrate inpainting with a pre-trained StyleGAN-2 generator trained on cats.Figure 7 identifies the first column as the never-observed original, the second as the observed image, and the third as the ILO reconstruction.Source: §7.4 · pp. 15, 17, Figure 7 · p. 17

7.5. Ethical Considerations

The paper discusses bias and potential misuse associated with expanding a generator’s range, and reports negative results and tuning difficulties. Its l1 analysis does not extend to an l2 dilation with the same theoretical result.

7.5. Ethical Considerations

  • Extending the generator range produces more diverse generations in the authors’ observations, though they encourage further research on bias.Prior work had reported biased StyleGAN generations.Source: §7.5 · p. 18
  • The authors say ILO makes DeepFakes easier to create and raise whether existing defenses remain robust to images outside the GAN range.Source: §7.5 · p. 18
  • Reference-image similarity combined with a robust classifier can be abused to generate offensive images; detecting and preventing abuse is left for future work.Source: §7.5 · p. 18

7.6. Things that did not work

  • Jointly optimizing all noise vectors produced blurry or unrealistic reconstructions, even when MSE to the unobserved image decreased.The authors attribute the failure to expanding generator space without sequential optimization and constraints making it too powerful.Source: §7.6 · p. 18
  • Moving to the next layer when observed MSE flattens can yield unrealistic reconstructions in deeper layers, where the authors use very small radii.Layer-wise learning rates, step counts, and radii are toilsome to tune and do not generalize across tasks, though they mostly generalize across images.Source: §7.6 · p. 18
  • The authors could not establish a similar theoretical result for l2 dilation because its measurement bound retains linear dependence on intermediate dimension.Whether an l2 result can be proved remains unclear; practice showed no significant difference between l1 and l2 projections.Source: §7.6 · p. 18

Related papers

Loading 2102.07364v1…