PaperlayerView exact revision on arXiv

Source-linked AI summary

Controlling generative models with continuous factors of variations

Antoine Plumerault, Hervé Le Borgne, Céline Hudelot

arXiv:2001.10238v1cs.LGcs.CVeess.IVstat.ML

TL;DR

Generative models offer photo-realistic images, but their use is limited by poor control and weak understanding of learned representations. The paper finds meaningful latent directions to control continuous image properties, and reports precise control of intuitive factors such as translation and scale in BigGAN. This is presented as a step toward understanding generative models’ learned representations.

  • Problem

    Generative models’ usefulness is often limited by lack of control over generated images and poor understanding of their learned representations.

    Source: §1 · p. 1

  • Method

    The method finds meaningful latent-space directions for controlling continuous image properties without labeled data or an encoder.

    Source: §1 · p. 2

  • Results

    The authors report that a BigGAN latent subspace can be interpreted through intuitive factors of variation, including translation and scale.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The authors describe interpreting latent directions as an important step toward understanding representations learned by generative models.

    Source: §5 · p. 10

    Unlucky initialization can make optimization-based inversion converge very slowly, and image transformation trajectories may yield small loss gradients because the natural-image manifold is highly curved.

    Source: §2.1.2 · p. 4

Abstract

from arXiv · show

Recent deep generative models are able to provide photo-realistic images as well as visual or textual content embeddings useful to address various tasks of computer vision and natural language processing. Their usefulness is nevertheless often limited by the lack of control over the generative process or the poor understanding of the learned representation. To overcome these major issues, very recent work has shown the interest of studying the semantics of the latent space of generative models. In this paper, we propose to advance on the interpretability of the latent space of generative models by introducing a new method to find meaningful directions in the latent space of any generative model along which we can move to control precisely specific properties of the generated image like the position or scale of the object in the image. Our method does not require human annotations and is particularly well suited for the search of directions encoding simple transformations of the generated image, such as translation, zoom or color variations. We demonstrate the effectiveness of our method qualitatively and quantitatively, both for GANs and variational auto-encoders.

1 INTRODUCTION

Although generative models can produce photo-realistic images, their use is often limited by poor control over generated content. This paper proposes finding latent directions for precise control of continuous image properties, without labeled data or an encoder.

1 INTRODUCTION

  • Generative models’ use is often limited by lack of control over generated images, despite their ability to produce high-resolution photo-realistic images.Source: §1 · p. 1
  • More control could let users choose specific properties of generated images for approaches that create new training examples.Source: §1 · p. 2
  • Studying latent-space directions can also provide insight into the structure of generative representations, including unsupervised representations.Source: §1 · p. 2
  • Prior control over generation often targeted discrete attributes and required labels and an encoder, while precise control of continuous parameters remained unavailable.The paper distinguishes discrete modal factors, such as object category, from continuous factors such as object size and position.Source: §1 · p. 2
  • The paper proposes finding meaningful latent-space directions that precisely control continuous factors such as vertical position, horizontal position, and scale, without labeled data or an encoder.Source: §1 · p. 2
  • The contributions include controlling image properties by sampling latent representations along linear directions, a reconstruction loss for inversion, analysis of inversion difficulty, and study of disentanglement’s impact on control.Source: §1 · p. 2

2 LATENT SPACE DIRECTIONS OF A FACTOR OF VARIATION

The section estimates latent directions for continuous image transformations by recovering transformed-image codes and relating transformation parameters to latent coordinates. It addresses inversion quality and convergence through a frequency-weighted loss and sequential optimization.

2 LATENT SPACE DIRECTIONS OF A FACTOR OF VARIATION

  • A transformation direction is estimated by comparing the latent codes of an original image and its transformed counterpart.The method frames the task as finding zT such that G(zT) approximates the transformed image, then uses the code difference to estimate the direction.Source: §2 · pp. 2–3

2.1 LATENT SPACE TRAJECTORIES OF AN IMAGE TRANSFORMATION

  • Transformed-image codes are recovered by minimizing reconstruction error, with a norm constraint motivated by typical Gaussian latent-code radii and the risk of low-likelihood solutions.Optimization can otherwise produce unrealistic reconstructions when codes land in low-likelihood regions.Source: §2.1 · p. 3

2.1.1 CHOICE OF THE RECONSTRUCTION ERROR L

  • Pixel-wise losses can produce blurry reconstructions, so the proposed Gaussian-filtered difference loss downweights high frequencies to retain more image detail.The loss is expressed as the norm of the image difference convolved with a Gaussian kernel.Source: §2.1.1 · p. 3

2.1.2 RECURSIVE ESTIMATION OF THE TRAJECTORY

  • Sequentially optimizing smaller transformation steps, each initialized from the previous solution, guides inversion along the manifold without training an auxiliary initializer.Undefined transformed-image regions are ignored, and the tenth of latent codes with the worst reconstruction errors are discarded in practice.Source: §2.1.2 · p. 4

2.2 ENCODING MODEL OF THE FACTOR OF VARIATION IN THE LATENT SPACE.

  • The transformation parameter is modeled as gθ(⟨z,u⟩), with u and θ fitted by minimizing error between observed and predicted transformation differences for paired latent codes.When the parameter distribution is unknown, the model uses trainable gθ; piecewise-linear functions are typical, with ||u|| = 1 and gθ(0) = 0.Source: §2.2 · p. 5
  • The learned mapping supports sampling images to match a chosen factor distribution and may help reveal potential bias in the training dataset through generated-output distributions.The paper distinguishes control over a single generated output from control over the distribution of outputs.Source: §2.2 · p. 5

3 EXPERIMENTS

Experiments assess control of position and scale on ILSVRC and dSprites, including whether BigGAN directions transfer across categories and how disentanglement affects control.

3 EXPERIMENTS

  • Experiments use dSprites and ILSVRC; the BigGAN latent vector has six parts, and β-VAEs are trained on dSprites for 1e5 steps.dSprites contains 737280 binary 64 × 64 images; ILSVRC contains 1.2M natural images across 1,000 categories.Source: §3 · p. 6

3.1 QUANTITATIVE EVALUATION METHOD

  • The quantitative evaluation estimates position from object barycenters and scale from salient-pixel proportion, then measures each factor’s standard deviation as latent codes move along the direction.For natural images, saliency detection produces the binary image used to estimate the barycenter; evaluation samples codes and generates images across t values.Source: §3.1 · p. 6

3.2 RESULTS ON BIGGAN

  • The method controls object position and scale relatively precisely on ten ILSVRC categories, and a common direction learned across trajectories also works on ten validation categories.Figure 2 reports the quantitative results; at scales near 1.0, apparent failure is likely due to poor saliency-model performance when the object nearly fills the image.Source: §3.2 · p. 6, Figure 2 · p. 7
  • Spatial-factor directions are encoded mainly in BigGAN’s first latent-code part, while vertical position has a larger level-five contribution than horizontal position or scale.The authors suspect this difference reflects correlations between vertical object position and the background, which changes with vertical translation.Source: §3.2 · p. 7

3.3 THE IMPORTANCE OF DISENTANGLED REPRESENTATIONS

  • On dSprites, moving in the latent-space direction found by the method controls object position, and larger β yields better results as the standard deviation decreases, enabling more precise positioning.Source: §3.3 · p. 7

4 RELATED WORKS

Related work studies latent-space control, generator inversion, and methods for interpreting or manipulating generated images. The paper distinguishes its approach by finding directions without direct labels or changes to model training.

Model families

  • The paper treats GAN-like models and auto-encoders as distinct families, respectively lacking and providing an encoder for obtaining image latent representations.Source: §4 · p. 7

Related approaches to control

  • Conditional GANs use labeled data for control, while InfoGAN learns disentanglement during training; this method instead finds directions in existing models without changing their learning process (Odena et al., 2016; Chen et al., 2016).Bau et al. (2018) analyze intermediate neuron activations to control object presence, whereas this work focuses on latent space.Source: §4 · p. 8

Generator inversion

  • Earlier generator-inversion methods optimized latent codes for reconstruction, but the authors report that these approaches fail on ILSVRC and introduce a reconstruction loss that significantly improves reconstruction quality (Creswell & Bharath, 2016).They also reason theoretically about why generator inversion is difficult.Source: §4 · p. 9

Closest related work

  • Compared with Goetschalckx et al. (2019) and Jahanian et al. (2019), this work trains on generated trajectories, uses a saliency model for broader evaluation, and supports auto-encoders as well as BigGAN.The authors identify their latent-space model as the main difference and claim it permits more precise control.Source: §4 · p. 9

5 CONCLUSIONS

The paper proposes extracting meaningful latent-space directions to control generated-image properties and reports that BigGAN’s linear subspace represents translation and scale.

5 CONCLUSIONS

  • A linear subspace of BigGAN’s latent space can be interpreted through intuitive image factors—translation and scale—an important step toward understanding generative-model representations.Source: §5 · p. 10

A PENALTY ON THE AMPLITUDE OF FREQUENCIES DUE TO MSE

The analysis explains how MSE’s frequency-domain contributions can favor smoother reconstructions when high-frequency phase is uncertain. In experiments, the authors use a convolutional β-VAE to generate 64×64 images.

A PENALTY ON THE AMPLITUDE OF FREQUENCIES DUE TO MSE

  • Under MSE, the Plancherel theorem expresses reconstruction error equivalently in the image and Fourier domains, allowing the loss contribution at a frequency to be examined.The analysis represents each image’s Fourier component by its magnitude and phase.Source: A PENALTY ON THE AMPLITUDE OF… · p. 13
  • Treating high-frequency phase as uniformly uncertain makes the expected high-frequency loss contribution depend on the generated image’s magnitude.The text assumes the generator may be unable to model every high-frequency pattern.Source: A PENALTY ON THE AMPLITUDE OF… · p. 13
  • Because the target-image magnitude term is constant during optimization, minimizing MSE tends to favor smaller generated high-frequency magnitudes and smoother images.The passage states that this preference means fewer high frequencies.Source: A PENALTY ON THE AMPLITUDE OF… · p. 13
  • For dSprites experiments, the authors use a convolutional β-VAE that generates 64×64 images, with a decoder built as the encoder’s transposed-convolution counterpart.Table 1 is identified as the β-VAE architecture used for those experiments.Source: B β-VAE ARCHITECTURE · p. 13, Table 1 · p. 13

C QUALITATIVE AND QUANTITATIVE EXPERIMENTS WITH OUR

The experiments compare reconstruction losses and latent-code constraints, and assess reconstructions with LPIPS. The proposed reconstruction error is reported to yield perceptually closer images than MSE or DSSIM, while the latent-norm constraint avoids artifacts in the illustrated example.

C QUALITATIVE AND QUANTITATIVE EXPERIMENTS WITH OUR

  • Figure 7 compares MSE, DSSIM, and the proposed loss with and without a latent-norm constraint; the unconstrained proposed loss shows artifacts.The text attributes the absence of artifacts in the constrained approach to restricting z to a ball of radius d.Source: Figure 7 · p. 14, C QUALITATIVE AND QUANTITATIVE… · p. 14
  • In the representative Figure 6 example, σ = 3 and σ = 5 give good reconstructions, while higher σ values reduce accuracy by penalizing low frequencies.The caption notes that a 3-pixel standard deviation was typically used for the kernel.Source: C QUALITATIVE AND QUANTITATIVE… · p. 14, Figure 6 · p. 14
  • MSE reconstructions have higher LPIPS standard deviation, which the authors say suggests some images are perceptually downgraded, particularly textured ones.The text refers to its frequency-domain discussion to explain the textured-image case.Source: C QUALITATIVE AND QUANTITATIVE… · pp. 14–15
  • The proposed reconstruction error produces images reported as perceptually closer to targets by LPIPS than MSE- or DSSIM-based reconstructions.The evaluation reconstructs one image from each of 1,000 ILSVRC categories with a 3,000-iteration budget; Table 2 reports the measurements.Source: C QUALITATIVE AND QUANTITATIVE… · p. 14, Table 2 · p. 15

D ON THE DIFFICULTY OF OPTIMIZATION ON THE NATURAL IMAGE MANIFOLD.

The section links the difficulty of latent-code optimization to the geometry of the natural-image manifold. For curved transformations such as translation and rotation, pixel-space descent can be nearly orthogonal to the manifold, making its latent gradient small.

D ON THE DIFFICULTY OF OPTIMIZATION ON THE NATURAL IMAGE MANIFOLD.

  • Figure 8 compares shortest pixel-space paths with transformation trajectories for translation, scale, and orientation, illustrating the curved walks considered in the analysis.The trajectories are derived from progressively transformed dSprites images and visualized on their two main PCA axes.Source: Figure 8 · p. 15, D ON THE DIFFICULTY OF OPTIMIZATION ON… · p. 15
  • For large translations, the shortest pixel-space direction is near orthogonal to the manifold; the same issue occurs for rotation, less so for scale, but not for linear brightness changes.The resulting reconstruction-loss gradient with respect to the latent code is small in near-orthogonal cases.Source: D ON THE DIFFICULTY OF OPTIMIZATION ON… · p. 15, Figure 8 · p. 15
  • When pixel-space descent is near orthogonal to the generative manifold, optimization slows and can stop if the loss gradient is orthogonal to it.The authors connect this behavior to the manifold’s curvature.Source: D ON THE DIFFICULTY OF OPTIMIZATION ON… · pp. 15–16
  • In an idealized GAN example encoding a circle’s position, translating a non-overlapping circle yields zero reconstruction gradient because a small move does not change the error.The example uses a white circle on black background and a two-dimensional latent space.Source: D ON THE DIFFICULTY OF OPTIMIZATION ON… · p. 16

E ADDITIONAL QUALITATIVE EXAMPLES

The additional BigGAN examples illustrate learned controls for object position, scale, and brightness across selected ILSVRC categories. Brightness control is not successful for some selected categories, which the authors suggest may lack dark images in their training data.

E ADDITIONAL QUALITATIVE EXAMPLES

  • Figure 9 presents qualitative results for ten ILSVRC categories across horizontal and vertical translation, scaling, and brightness.The examples use the BigGAN model.Source: Figure 9 · p. 16, E ADDITIONAL QUALITATIVE EXAMPLES · p. 17
  • The authors report position and scale controls for the selected categories, but brightness control fails for some; they suggest those categories may lack dark training images.They learn position and scale directions on ten categories, and brightness directions on the five top categories.Source: E ADDITIONAL QUALITATIVE EXAMPLES · p. 17

F QUALITATIVE COMPARISON BETWEEN OUR OPTIMIZATION METHOD AND THE NAIVE METHOD.

Figure 10 compares convergence speed for the proposed and naive optimization methods on a single example.

F QUALITATIVE COMPARISON BETWEEN OUR OPTIMIZATION METHOD AND THE NAIVE METHOD.

  • Figure 10 compares single-example convergence speed for the proposed method (equation 5, top) and naive approach (equation 4, bottom), with optimization steps shown under Adam at learning rate 1e−1.Source: Figure 10 · p. 17
Loading 2001.10238v1…