PaperlayerView exact revision on arXiv

Source-linked AI summary

On the "steerability" of generative adversarial networks

Ali Jahanian, Lucy Chai, Phillip Isola

arXiv:1907.07171v4cs.CVcs.LG

TL;DR

GANs can generate realistic images, but whether they model the visual world comprehensively remains unresolved. This paper learns and evaluates latent-space trajectories for visual transformations, finding useful but finite distributional shifts whose range reflects training-data variability and can be expanded by augmentation and joint optimization.

  • Problem

    The paper asks whether GANs can generalize beyond their training distribution by producing navigable visual transformations rather than merely reproducing training examples.

  • Method

    The authors learn self-supervised latent-space trajectories for image transformations, quantify their relationship to training-data variability, and test data augmentation with joint generator-and-trajectory optimization.

  • Results

    GANs can steer generated images through camera zoom, movement, rotations, and recolorization, but naive distribution shifts remain finite and cannot leave the training-data support entirely.

  • Takeaways & Limitations

    Realistic distributional shifts are possible, but their extent depends on sufficient dataset diversity along the transformed dimension; augmentation and joint optimization can increase steerability.

  • Takeaways & Limitations

    Steering remains constrained by biases and limited variability in the training data, with transformations unable to extrapolate entirely outside the dataset’s support.

Abstract

from arXiv · show

An open secret in contemporary machine learning is that many models work beautifully on standard benchmarks but fail to generalize outside the lab. This has been attributed to biased training data, which provide poor coverage over real world events. Generative models are no exception, but recent advances in generative adversarial networks (GANs) suggest otherwise - these models can now synthesize strikingly realistic and diverse images. Is generative modeling of photos a solved problem? We show that although current GANs can fit standard datasets very well, they still fall short of being comprehensive models of the visual manifold. In particular, we study their ability to fit simple transformations such as camera movements and color changes. We find that the models reflect the biases of the datasets on which they are trained (e.g., centered objects), but that they also exhibit some capacity for generalization: by "steering" in latent space, we can shift the distribution while still creating realistic images. We hypothesize that the degree of distributional shift is related to the breadth of the training data distribution. Thus, we conduct experiments to quantify the limits of GAN transformations and introduce techniques to mitigate the problem. Code is released on our project page: https://ali-design.github.io/gan_steerability/

1 INTRODUCTION

This paper studies whether GANs can be navigated through latent-space directions that produce realistic visual transformations. It finds that steering enables several transformations but remains constrained by training-data variability, while augmentation and joint optimization expand the steerable range.

  • 1 INTRODUCTION: Qualitative examples show successful navigation such as zooming toward a dog, but continued zooming and moving a daisy out of frame quickly reach limits.The examples illustrate both realistic steering and failure to extend the transformation indefinitely.
  • 1 INTRODUCTION: Steering can shift generated-image distributions only within limits: transformations cannot extrapolate entirely beyond the training dataset’s support.The achievable shift is related to variability in the corresponding training-data attribute.
  • 1 INTRODUCTION: GAN latent-space walks can produce camera motion and color transformations without labeled attributes or paired source and target images.The walks are learned self-supervised, using generated images rather than labeled edits.
  • 1 INTRODUCTION: Linear latent walks are as effective as more complex nonlinear walks, suggesting that GANs roughly linearize these image operations without explicit training for them.The comparison concerns the effectiveness of the walk trajectories, not a claim that all transformations are globally linear.
  • 1 INTRODUCTION: The learned transformations generalize across BigGAN, StyleGAN, and DCGAN, whose latent spaces exhibit different disentanglement properties.The framework is presented as general-purpose across these architectures.
  • 1 INTRODUCTION: Data augmentation and jointly training the latent walk with generator weights increase the attainable transformation range.Joint optimization also decreases the L2 error relative to frozen-generator objectives.

2 RELATED WORK

The related work situates this paper among latent-space editing, dataset-bias, generative-content, and downstream image-manipulation research. Its distinction is self-supervised trajectory learning without labeled attributes or paired source and target images.

  • Interpolations in latent space: Earlier GAN editing methods learned linear directions for labeled attributes, while flow-based models edited encoded images through interpolation between source and target representations.These approaches use attribute labels or distinct images to define edits.
  • Interpolations in latent space: This paper instead learns latent-space trajectories self-supervised on individual generated source images, without labeled attributes or distinct source and target images.The contrast identifies the paper’s methodological position within prior latent-space editing work.
  • Dataset bias: Prior work links dataset and architectural biases to model generalization, including biases from canonical photographic views that models may inherit.The paper’s focus is on biases in generative models originating from their training data.
  • Generative models for content creation: Generative-model content-creation work enables fine-tuning, whereas this paper presents image modification through a single control knob: transformation magnitude in latent space.The control changes image properties by varying the magnitude of the learned transformation.
  • Applications of latent space editing: Related image-editing applications include measuring biases in facial-attribute detectors and editing face attributes such as age and expression.These applications differ from this paper’s study of training-data biases in the generative model.

3 METHOD

The method learns latent-space trajectories that make generated images undergo specified edits, then measures steerability and explores augmentation and joint optimization to extend transformation ranges.

  • Latent-space transformations: GANs map latent codes z to outputs x, so the method seeks output-space transformations by moving through latent space.This frames latent navigation as an equivariance-like relationship between latent movements and image transformations.
  • Latent-space transformations: The linear walk learns a vector w whose scaled step α controls transformation magnitude, producing G(z + αw) to approximate edit(G(z), α).The objective uses image distance between the walked image and the edited target, with L2 loss and similar results under LPIPS.
  • Learning trajectories: The method learns walks self-supervised by editing arbitrary generated images, without labeled attributes or distinct source and target images.The target edit is generated from the source image, and the walk is optimized to reduce their distance.
  • Latent-space transformations: Nonlinear trajectories learn a position-dependent function f and apply it recursively, approximating larger transformations through repeated small steps.The function is composed n times, while a neural network parametrizes f(z).
  • Quantifying steerability: Steerability is quantified by comparing dataset attribute distributions with distributions after latent-space walks, using sampled pixels for color and detected object boxes for zoom and shift.Color evaluation samples 100 pixels per image; object transformations use MobileNet-SSD bounding boxes.
  • Reducing transformation limits: To reduce transformation limits, the authors augment training images with the corresponding edits and jointly optimize generator weights with a linear walk vector.The combined edit and GAN objectives encourage linear organization and, with augmentation, larger transformation ranges.

4 EXPERIMENTS

Experiments show that latent-space walks can produce diverse visual transformations, but their magnitude is limited by realism and dataset-dependent biases. Broader training-data variability supports wider distribution shifts, while augmentation and joint generator-walk training improve steerability.

  • 4.1 WHAT IMAGE TRANSFORMATIONS CAN WE ACHIEVE IN LATENT SPACE?: BigGAN experiments learn shared latent-space walks that transform image position, zoom, color, and brightness across ImageNet classes.The study also tests StyleGAN and nonlinear walks to examine architectural and trajectory alternatives.
  • 4.1 WHAT IMAGE TRANSFORMATIONS CAN WE ACHIEVE IN LATENT SPACE?: Increasing transformation magnitude eventually makes images unrealistic or stops changing them further.For zoom, LPIPS distance decreases near the limits, indicating that generated images converge; zooming beyond 8x increases FID.
  • 4.1 WHAT IMAGE TRANSFORMATIONS CAN WE ACHIEVE IN LATENT SPACE?: A shared latent direction affects classes differently: jellyfish and sports cars can be recolored, whereas goldfinches and firetrucks remain largely constrained by their usual colors.The reported class comparisons are statistically separated: t = 20.77, 8.14, and 6.84, each with p < 0.001.
  • 4.1 WHAT IMAGE TRANSFORMATIONS CAN WE ACHIEVE IN LATENT SPACE?: Shift transformations move the center-object distribution between 0.3 and 0.7 of image width or height, but vertical shifts beyond 150 pixels produce unrealistic images.The transformed distributions show decreasing intersection and converging modes as transformation magnitude increases.
  • 4.2 HOW DOES THE DATA AFFECT THE TRANSFORMATIONS?: Transformation capacity correlates positively with per-class dataset variability: low-spread robin images permit smaller zoom shifts than broad-spread laptop images.The slope of every observed trend differs significantly from zero, with p < 0.001 for all transformations.
  • 4.3 ALTERNATIVE ARCHITECTURES AND WALKS: Data augmentation improves steerability, and jointly optimizing generator weights with the walk further reduces L2 error while enabling larger effects.Compared with the vanilla frozen-generator setup, augmentation preserves structural integrity at larger transformation magnitudes.

5 CONCLUSION

The paper finds that GANs can be steered to simulate several simple image transformations, but their ability to shift generated distributions remains finite. Data augmentation and joint optimization increase steerability, while training-data bias constrains generalization.

  • GANs can simulate camera zoom, horizontal and vertical movement, camera rotations, and recolorization through latent-space walks.
  • The generated distribution can be transformed only to some degree and cannot be extrapolated entirely outside the training data’s support.
  • Data augmentation and joint optimization of the model and walk trajectory increase the range of images the models can create.

A METHOD DETAILS

The walk vector is learned from 20,000 latent samples and shared across all ImageNet categories for BigGAN.

  • 20,000 unique latent-space samples are used to learn the walk vector with mini-batch stochastic gradient descent and Adam.
  • The optimization is implemented in TensorFlow.
  • The learned vector w is shared across all ImageNet categories for BigGAN.

A.2 IMPLEMENTATION DETAILS FOR LINEAR WALK

The linear-walk implementation learns latent directions for shifts, zoom, color, and rotations by matching transformed source images, with α controlling transformation magnitude.

  • Each transformation is learned without direct supervision by applying the desired edit to source images, while α controls the transformation extent.
  • Shift: Shift walks model horizontal and vertical movement using normalized pixel displacement and masks losses to visible image regions.
  • Zoom: Zoom walks model inward and outward changes up to four times the original image, using logarithmic scaling and background inpainting masks.
  • Color: Color walks represent RGB changes with three independently learned latent directions combined as z + αRwR + αGwG + αBwB.
  • Rotation: 2D and simulated 3D rotations are trained over −45 to 45 degrees with normalized step sizes and masks on visible target regions.

A.3 LINEAR NN(z) WALK

A latent walk parameterized as NN(z) cannot learn input-dependent directions when arbitrary step sizes enforce compositional consistency. The nonlinear alternative uses fixed discrete steps to avoid this constraint.

  • An NN(z)-parameterized walk is intended to map each latent code to a desired transformed code after a variable-sized step.
  • Because arbitrary α values require one 2α step to equal two α steps, the learned function becomes a linear trajectory independent of z.
  • The nonlinear walk instead uses discrete fixed step sizes and defines F(z) as z+NN(z), followed by latent-magnitude renormalization.
  • The discrete-step approach mimics Euler integration and avoids optimizing the equality imposed by arbitrary step sizes.
  • Positive and negative directions use separate two-layer neural networks, with the step size chosen to match the linear trajectory within 4–5 steps.

B.1 MODEL AND DATA DISTRIBUTIONS

The analysis compares generated and training-data distributions, then examines how latent walks alter images and how those changes are measured. Perceptual change alone is insufficient to establish transformation accuracy or naturalness.

  • Model and data distributions: Generated-image attribute distributions are compared with ImageNet training-data statistics to assess how well the model matches dataset variability.The attributes include color luminance, object bounding-box size, and bounding-box center.
  • Transformation limits: Increasing latent transformation magnitude makes shift and zoom changes diminish perceptually while generated images become unrealistic.For shift and zoom, consecutive-step LPIPS distances decrease as the magnitude of α grows.
  • Transformation limits: Color and 2D/3D rotation maintain a steadier transformation rate as α increases.These operations differ from shift and zoom in their perceptual-distance behavior.
  • Measurement: Attribute detectors, including object bounding boxes, are used to evaluate whether the intended transformations are achieved.The perceptual-distance analysis measures image change, not transformation-specific accuracy or natural-looking outputs.
  • Measurement: The latent walk is learned by minimizing an objective function, with Euclidean loss used for the main analysis and LPIPS used for comparison.The LPIPS walks use the same training setup as the linear-L2 walks.

B.4.2 NON-LINEAR WALKS

The paper extends linear latent walks to nonlinear trajectories and evaluates them across BigGAN and StyleGAN-related generators. Nonlinear walks preserve distribution-shifting trends but can reduce realism, while StyleGAN’s intermediate W space supports strong and localized edits.

  • Nonlinear walks: A learned neural function F(z) performs a discrete ε-step transformation on latent code z, producing nonlinear walks.The nonlinear objective replaces continuous steps with discrete step sizes.
  • Nonlinear walks: Nonlinear walks reproduce the main trend: latent walks modify generated distributions, with transformation extent related to dataset variability.These experiments use the same general quantitative analyses as the main text.
  • Nonlinear walks: Nonlinear transformations produce greater FID increases, indicating larger deviations from natural-looking images.The reported increase suggests nonlinear outputs look less realistic than the corresponding natural images.
  • StyleGAN experiments: StyleGAN experiments learn linear walk vectors in the intermediate W space because it offers improved attribute disentanglement.The experiments cover color, shift, and zoom on cat, car, and face datasets.
  • StyleGAN experiments: Progressive GAN requires substantially more training samples than StyleGAN W space to obtain comparable visible color changes.StyleGAN achieved drastic changes with as few as 2k samples, whereas Progressive GAN used 60k without an equally strong effect.
  • StyleGAN experiments: StyleGAN W-space color edits can be localized to segmented car regions, suggesting object-level disentanglement.The model roughly confines the color transformation to the specified segmented region.

B.8 ADDITIONAL RESULTS FOR IMPROVING MODEL STEERABILITY

Additional experiments test ways to improve steerability and visualize learned transformations across generators, datasets, objectives, and combined edits. The supplied results emphasize augmentation and joint training as mechanisms for increasing transformation effects.

  • Improving model steerability: Data augmentation improves steerability, and jointly training walk trajectories with generator weights enables larger transformation effects.These approaches expose the discriminator to edited real images or encourage the generator to learn linear walks.
  • Distribution comparisons: Figure 10 compares generated and training-data statistics for luminance, bounding-box size, and bounding-box center.The comparison targets color, zoom, and shift properties.
  • Transformation measurements: Figure 11 measures LPIPS distances between consecutive transformation steps, showing mean variation, one standard deviation, and individual samples.The experiments sample 1000 BigGAN images from randomly selected categories.
  • Transformation measurements: Figures 12 and 13 compare object bounding boxes in ImageNet images with boxes produced under generated zoom and horizontal or vertical shifts.The generated-image examples use randomly selected transformation magnitudes.
  • Trajectory objectives: Figures 14–19 compare linear L2, linear LPIPS, and nonlinear trajectories using qualitative walks and quantitative analyses.The nonlinear experiments combine L2 for color with LPIPS for the remaining transformations.
  • Specialized transformations: Figures 28–30 examine latent-space choices, segmented color targets, contrast walks, and jointly or separately learned combinations of zoom and shifts.The combined-edit visualization includes target images, joint walks, and combinations of separately trained walks.
  • MNIST evaluation: Figures 31–33 evaluate zoom, horizontal shift, and 2D rotation steerability with an MNIST DCGAN.Odd rows show target images and even rows show learned transformations.
Loading 1907.07171v4…