Source-linked AI summary
Seeing What a GAN Cannot Generate
David Bau, Jun-Yan Zhu, Jonas Wulff, William Peebles, Hendrik Strobelt, Bolei Zhou, Antonio Torralba
TL;DR
Mode collapse can omit meaningful portions of a GAN’s target distribution, while existing tools offer limited insight into which concepts are missing. The paper compares segmentation statistics and uses layer inversion to visualize omissions, finding that difficult object classes may be omitted entirely even in visually high-quality outputs.
Problem
Mode collapse is a major GAN challenge, yet little work identifies or explains which modes and semantic concepts a model omits.
Method
The paper compares segmented object distributions in generated and training images, then uses Layer Inversion to reconstruct real images through an intermediate generator layer.
Results
Across recent GANs and scene datasets, difficult object classes such as people and fences are often omitted entirely rather than rendered poorly or noisily.
Takeaways & Limitations
Distribution statistics and layer inversions reveal semantic objects, parts, and styles that GAN generators cannot represent.
Takeaways & Limitations
The proposed methods are a first step toward understanding GAN latent spaces, leaving the causes of mode dropping and ways to address it for future work.
Abstract
from arXiv · showhide
Despite the success of Generative Adversarial Networks (GANs), mode collapse remains a serious issue during GAN training. To date, little work has focused on understanding and quantifying which modes have been dropped by a model. In this work, we visualize mode collapse at both the distribution level and the instance level. First, we deploy a semantic segmentation network to compare the distribution of segmented objects in the generated images with the target distribution in the training set. Differences in statistics reveal object classes that are omitted by a GAN. Second, given the identified omitted object classes, we visualize the GAN's omissions directly. In particular, we compare specific differences between individual photos and their approximate inversions by a GAN. To this end, we relax the problem of inversion and solve the tractable problem of inverting a GAN layer instead of the entire generator. Finally, we use this framework to analyze several recent GANs trained on multiple datasets and identify their typical failure cases.
1. Introduction
The paper asks how to identify what GANs cannot generate, addressing mode collapse beyond single-number quality metrics. It proposes distribution- and instance-level analyses that expose omitted semantic objects and specific reconstruction failures.
- Motivation: Mode collapse omits portions of the target distribution, but existing tools provide little insight into which concepts state-of-the-art GANs cannot generate.The paper focuses on explaining omissions rather than only measuring distance between generated and target distributions.
- Approach: The framework analyzes scene generators at two levels: distributions of segmented object classes and individual image instances.Scenes are used because images can be parsed into clear semantic components through segmentation.
- Distribution-level analysis: Generated Image Segmentation Statistics compare segmented generated and training images to identify object classes underrepresented in the generated distribution.In church models, people, cars, and fences occupy fewer generated pixels than in the training distribution.
- Instance-level analysis: Layer Inversion finds reconstructions by inverting a generator layer instead of the entire GAN, making reconstruction tractable for complex state-of-the-art models.Differences between real images and reconstructions reveal features and objects the generator cannot faithfully draw.
- Findings: Across recent GANs and scene datasets, omitted classes are often absent rather than distorted or rendered noisily, even when average visual quality is high.Examples include large human figures and parallel fence lines being skipped entirely.
2. Related work
Prior GAN research emphasizes evaluation metrics, inversion, and visualization of learned features or capabilities. This work instead complements those approaches by diagnosing semantic concepts that generators fail to capture.
- Evaluation: Existing unconditional GAN evaluation includes Inception Score, Fréchet Inception Distance, and Wasserstein Sliced Distance for quantifying model quality or differences.The related work frames evaluation and comparison as important issues in generative modeling.
- Mode-collapse analysis: The paper positions its tools as explanations of mode collapse that complement existing single-number metrics and reveal model limitations.Its emphasis is diagnosis rather than replacing quantitative evaluation.
- Network inversion: Prior inversion studies support photo manipulation, feature learning, high-precision inversion, and analysis of generator limitations in simpler models.The paper distinguishes its setting from prior work on five-layer DCGAN generators and other specialized inversions.
- Network visualization: Unlike visualization work focused on discriminators or learned internal semantics, this paper examines semantic concepts a GAN generator fails to capture.The perspective is complementary to understanding what a GAN has learned.
3. Method
The method detects GAN omissions at distribution and instance levels. It compares segmented object statistics between generated and target images, then uses layer inversion to reconstruct real images through a tractable later-layer search.
- Distribution-level analysis: Generated Image Segmentation Statistics compare object-class distributions in generated and target images to identify omitted semantic classes.Images are segmented into 336 object classes, and mean pixel counts plus covariance statistics are collected across samples.
- Distribution-level analysis: The method uses segmentation statistics to expose systematic scene deviations, such as underrepresented curtains in generated bedrooms.Object-level deviations provide concrete classes for inspecting generator failures.
- Distribution-level analysis: Fréchet Segmentation Distance summarizes differences between generated and training segmentation statistics using their means and covariances.The experiments compare statistics from 10,000 generated samples with 10,000 natural images.
- Instance-level analysis: Layer Inversion finds image instances containing omitted classes and projects them onto images producible by the generator’s later layers.This compares a real image x with a reconstruction x′ generated from an intermediate representation.
- Instance-level analysis: Instead of solving difficult full-generator inversion, the method decomposes G into early layers and later layers Gf, then searches for an intermediate representation r∗.Because range(G) is contained in range(Gf), omissions found in Gf also imply omissions of G.
- Instance-level analysis: The inversion procedure first trains layer-wise approximate inverse networks, composes them into E, and optionally fine-tunes the composed network jointly.The resulting estimate z0 and intermediate representation r0 initialize the later layer-wise optimization.
4. Results
Across scene datasets, segmentation statistics expose distributional omissions, while layer-wise inversion visualizes the corresponding instance-level failures. The inversion method is highly accurate within the generator’s range, supporting reliable identification of images the generator cannot render.
- Generated Image Segmentation Statistics: StyleGAN matches LSUN bedroom segmentation statistics better than Progressive GAN, while WGAN-GP matches them least closely.Fréchet Segmentation Distance summarizes these distributional differences.
- Generated Image Segmentation Statistics: Finite-sample comparisons of 10,000-image subsets show small segmentation-statistics differences caused by sampling error.The sensitivity test compares two randomly selected LSUN bedroom subsets rather than a GAN with the true distribution.
- Identifying dropped modes: Progressive GAN bedroom outputs omit many object classes, including people, cushions, carpets, lamps, and several furniture types.Figure 5 pairs object-distribution comparisons with reconstructions and segmentations of photographs.
- Identifying dropped modes: The generators render fewer people than the target distribution and wholesale omission, rather than low-quality rendering, explains these object gaps.Large human figures and other difficult object classes may be absent entirely from reconstructions.
- Layer-wise inversion vs other methods: On 100 GAN-generated church images, the full layer-wise method achieves nearly perfect reconstructions, whereas baseline and ablation methods perform worse.The comparison evaluates mean correlations for z, layer4, and image pixels; the layered E then r variant is the full method.
- Layer-wise inversion vs other methods: Layer-wise network inversion alone reaches 95.5% correlation for intermediate features despite inaccurate latent-code recovery.The full method further optimizes an intermediate representation rather than relying only on the initial latent estimate.
- Layer-wise inversion vs other methods: Relaxed inversion accuracy above 99% makes large differences between real images and reconstructions unlikely to result from inversion failure.This supports using the method to identify images outside the generator’s representational range.
- Layer-wise inversion across domains: For LSUN church images, some architectural styles are dropped even when large-scale geometry is preserved; indoor inputs become outdoor-like, while outdoor inputs retain more detail.These results extend the qualitative analysis beyond bedroom generators.
5. Discussion
The paper presents methods to measure and visualize mode-dropping in state-of-the-art generative models, while framing them as an initial step toward understanding GAN latent spaces.
- Generated Image Segmentation Statistics compare labeled distributions to reveal semantic differences in generator output spaces.
- Layer inversions compare natural photographs with imperfect reconstructions to expose objects and styles that generators cannot represent.
- The proposed methods identify specific objects, parts, and styles that a generator cannot produce.
- The methods are described as a first step, leaving questions about ignored classes, training-set balance, and architectural effects for future work.
S.1. Supplemental Materials
The supplemental-materials passage provides the paper’s arXiv identifier and posting date.
- The paper is identified as arXiv:1910.11626v1.
- The listed posting date is 24 Oct 2019.
- The identifier and date appear in the computer-vision arXiv record.
S.1.1. Sensitivity measure
The sensitivity analysis examines sampling variability in Generated Image Segmentation Statistics and reports measurements using a table and figure.
- Generated Image Segmentation Statistics vary when the data is resampled because they are computed from sample statistics.
- Larger sample sizes can reduce the sampling error of these measurements.
- Figure S.1 and Table S.1 estimate sampling error at the 10,000-sample size used in the paper.
- Table S.1 reports measured sensitivity in Fréchet Segmentation Distance.
S.1.2. Analysis of unseen classes for additional GAN
The supplemental analysis gives additional examples of differences between generated and target semantic classes for Progressive GAN scene models.
- Figure S.2 analyzes a Progressive GAN trained on kitchens.
- Figure S.3 analyzes a Progressive GAN trained on living rooms.
- Figure S.4 analyzes a Progressive GAN trained on dining rooms.
S.1.3. Additional qualitative results on inversion
Additional reconstructions extend the qualitative comparison of inversion methods across generated and holdout images. The examples show that dropped details expose objects and styles the GAN cannot render, while semantic-distribution analyses identify omitted categories in several LSUN room datasets.
- The section expands inversion-method comparisons beyond Figure 5 with more reconstructions of generated and LSUN validation images.The additional examples cover church and bedroom models and compare several methods.
- Comparing two random 10,000-image subsamples estimates the finite-sampling measurement noise in Generated Image Segmentation Statistics.An infinite-sized sample would show no differences between the subsamples.
- Chairs, stove exhausts, and other objects are underrepresented in Progressive GAN images trained on LSUN kitchens.
- Coffee tables and ottomans are omitted from some Progressive GAN living-room images.
- Dining rooms containing kitchens lose many details in Progressive GAN images trained on LSUN dining rooms.
- Dropped details in reconstructions of natural photographs reveal objects and styles that the GAN cannot render.GAN-generated images can be reconstructed nearly perfectly, whereas natural photographs expose missing details.