PaperlayerView exact revision on arXiv

Source-linked AI summary

Diverse Image Generation via Self-Conditioned GANs

Steven Liu, Tongzhou Wang, David Bau, Jun-Yan Zhu, Antonio Torralba

arXiv:2006.10728v2cs.CVcs.LG

TL;DR

Unconditional GANs can miss modes, while class-conditional GANs encourage coverage but typically require labels. Self-conditioned GANs derive cluster labels from discriminator features and train a conditional GAN to cover the clusters; the method recovers modes on benchmarks and improves diversity and quality metrics on ImageNet and Places365.

  • Problem

    Unconditional GANs can miss modes in multimodal data, while class-conditional GANs encourage semantic coverage but often depend on costly labels.

    Source: §1 · p. 1, §3 · p. 2

  • Method

    Self-conditioned GANs periodically cluster discriminator features to create labels for a class-conditional GAN, explicitly encouraging coverage of the resulting partitions.

    Source: §1 · p. 2, §3 · p. 2

  • Results

    The method recovers modes on standard benchmarks and improves diversity and standard quality metrics on ImageNet and Places365 compared with previous unsupervised methods.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The method performs well even when synthesized labels outnumber data modes and generates qualitatively more diverse images than many unconditional GAN methods.

    Source: §5 · p. 8

Abstract

from arXiv · show

We introduce a simple but effective unsupervised method for generating realistic and diverse images. We train a class-conditional GAN model without using manually annotated class labels. Instead, our model is conditional on labels automatically derived from clustering in the discriminator's feature space. Our clustering step automatically discovers diverse modes, and explicitly requires the generator to cover them. Experiments on standard mode collapse benchmarks show that our method outperforms several competing methods when addressing mode collapse. Our method also performs well on large-scale datasets such as ImageNet and Places365, improving both image diversity and standard quality metrics, compared to previous methods.

1. Introduction

Unconditional GANs can miss modes in multimodal data, while class-conditional GANs encourage coverage but require costly labels. Self-conditioned GANs derive clusters from discriminator features and improve diversity and quality across benchmark and large-scale image generation tasks.

1. Introduction

  • Unconditional GANs can miss important modes in multimodal data, whereas class-conditional GANs encourage coverage of semantic categories but often require expensive labels.The passage gives omitted digits and object categories as examples of missed modes.Source: §1 · pp. 1–2
  • Self-conditioned GANs cluster discriminator features to derive labels and alternate between improving the representation and training a generator to cover discovered clusters.The partition is intended to reveal modes the generator is missing and explicitly require coverage of them.Source: §1 · p. 2
  • The method recovers modes on standard benchmarks and improves FID, Fréchet Segmentation Distance, and Inception Score on ImageNet and Places365 versus previous unsupervised methods.The benchmarks include mixtures of Gaussians, stacked MNIST, and CIFAR-10.Source: §1 · p. 2
  • The authors make their code and models available on their website.Source: §1 · p. 2

2. Related Work

Prior work addresses mode collapse through changes to adversarial training, latent recovery, or mixtures of generators, while conditional GANs for large datasets often rely on labels. This work instead clusters discriminator features to obtain labels and train toward diverse generation.

2. Related Work

  • For complex datasets such as ImageNet, state-of-the-art GANs are class-conditional and require ground-truth labels; a RotNet-based semisupervised approach uses labels for only a subset.The authors describe their method as fully unsupervised and general-purpose, and report outperforming the RotNet-based baseline in Section 4.4.Source: §2 · p. 2
  • During optimization, a slowly changing discriminator can guide the generator toward a point that maximizes the discriminator, contributing to mode collapse.The passage characterizes mode collapse as a fundamental GAN training challenge.Source: §2 · p. 2
  • Prior mode-collapse methods modify the loss, recover latent distributions, add auxiliary losses, pack discriminator inputs, or combine generators with diversity objectives.Unlike these approaches, this method partitions the real distribution and assigns a class-conditioned discriminator to each partition.Source: §2 · p. 2
  • Other clustering-based conditional GANs use unsupervised features or pretrained classifiers, whereas this method clusters discriminator features already present in GANs.The authors report simpler training and higher-quality generation than these approaches in their experiments.Source: §2 · p. 2

3. Method

The method dynamically partitions real data using discriminator features, then trains a class-conditional GAN on those clusters without ground-truth labels. Periodic reclustering updates the partitions as the discriminator learns, while matching preserves cluster identities across updates.

3. Method

  • Clustering the real data into dynamically determined partitions and conditioning the GAN on them encourages generated outputs to cover all partitions without ground-truth labels.The method addresses mode collapse, where generator support can be smaller than real-data support.Source: §3 · p. 2
  • Training begins by clustering samples in the randomly initialized discriminator feature space, then periodically updates clusters using current discriminator features.The class-conditional generator and discriminator are trained using the current partition between reclustering steps.Source: §3 · pp. 2–3
  • For each cluster, the conditional discriminator distinguishes its real images from generated images, while the corresponding generator learns to mimic that cluster.Cluster indices are sampled in proportion to cluster size, and latent inputs are sampled from a standard normal distribution.Source: §3 · p. 3, Figure 2 · p. 3
  • After reclustering, previous cluster means initialize the new clustering, and Hungarian min-cost matching aligns new and old clusters to minimize missing samples.Matching avoids retraining the conditional generator and discriminator from scratch.Source: §3 · pp. 3–4
  • The online k-means variant performs comparably on mode-collapse benchmarks but worse on real-image datasets, potentially because frequent updates destabilize training.Source: §3 · p. 4

4. Experiments

Experiments evaluate self-conditioned GANs on synthetic and image benchmarks, including large-scale datasets, with measures of diversity, image quality, reconstruction, and clustering. Across these settings, the results examine both the method’s performance and the stability of its clustering choices.

4.1. Experiment Details

  • Experiments use conditional GAN architectures with cluster labels, default to 100 clusters, and periodically recluster; large-scale runs use R1 regularization with γ = 0.1.The generator receives a learned class embedding, while the discriminator masks its cluster outputs according to the input label.Source: §4.1 · p. 4

4.2. Synthetic Data Experiments

  • On synthetic benchmarks, the method generates higher-quality and more diverse samples than competing unsupervised methods, while random labels fail to capture both.The online clustering variant also performs well, achieving almost perfect performance.Source: §4.2 · p. 4, Table 1 · p. 5

4.3. Stacked-MNIST and CIFAR-10 Experiments

  • On Stacked-MNIST and CIFAR-10, the method improves FID and IS over supervised class-conditional GANs; performance is robust across a range of k but worsens at extremes.A k that is too low approaches unconditional-GAN behavior, while a k that is too high can destabilize clustering and hurt GAN training.Source: §4.3 · p. 5, Table 2 · p. 5, Table 3 · p. 5

4.4. Large-Scale Image Datasets Experiments

  • The large-scale experiments train on 1.2 million ImageNet images and 1.8 million Places365 images without revealing class labels to the model.Both datasets use k = 100.Source: §4.4 · p. 5

4.4.1 Generation Realism and Diversity

  • On Places365 and ImageNet, the method improves quality and diversity over previous methods but does not reach fully supervised class-conditional quality.The reported metrics include FID, FSD, and IS; the tested unsupervised baselines are all outperformed.Source: Table 4 · p. 6, §4.4.1 · p. 6

4.4.2 Quantifying Mode Dropping with Image Reconstruction

  • On Places365, average reconstruction LPIPS is 0.433 for the method versus 0.528 for the baseline.Visualizations also show improved reconstruction of distinctive features during training, while vanilla GAN reconstructions struggle on most real images.Source: §4.4.2 · pp. 6–7, Figure 3 · p. 6

4.5. Clustering Metrics

  • Clusters often capture object categories rather than true classes, explaining low clustering scores, although the scores exceed random clustering.The reported clustering measures are NMI and purity.Source: §4.5 · p. 7, Table 5 · p. 7

4.6. Ablation Studies on Cluster Initialization and Matching

  • Without R1 regularization, cluster initialization and matching are needed for stable clustering and GAN quality; with R1, the method still works without them, but CIFAR-10 results worsen.The authors hypothesize that R1 stabilizes GAN training but can reduce discriminator feature quality and thereby worsen clusters.Source: §4.6 · p. 8, Table 6 · p. 8

5. Conclusion

Self-conditioned GANs reduce mode collapse by training a conditional GAN with labels clustered from discriminator features, and perform well on large-scale image generation.

5. Conclusion

  • Clustering-derived labels make conditional GANs effective at reducing mode collapse, outperforming several previous approaches.The method also performs well when the number of synthesized labels exceeds the number of data modes, and improves standard measures on ImageNet and Places365 while generating qualitatively more diverse images than many unconditional GAN methods.Source: §5 · p. 8

Appendix A. Additional Experiment details

The appendix specifies dataset-specific classifiers, clustering procedures, and evaluation settings, and reports that online clustering performs well on easier datasets but not convincingly on real images.

Appendix A. Additional Experiment details

  • Experiments use one Titan Xp GPU except ImageNet and Places365, which use four; synthetic Gaussian MGAN experiments set β = 0.125 following.Source: Appendix A · p. 11
  • Synthetic, Stacked-MNIST, and CIFAR experiments use classifiers for reverse-KL, with nearest-mode assignment for 2D data and pretrained classifiers for image data.Stacked-MNIST channel predictions form a label from 0 to 999; CIFAR-10 uses a pretrained CIFAR-10 classifier.Source: Appendix A · p. 11
  • ImageNet diversity visualizations use a pretrained torchvision ResNet-50, while Places visualizations use a pretrained ResNet-50 released by.Source: Appendix A · p. 11
  • Clustering uses discriminator last-hidden-layer features, with subsets of 10,000 synthetic points, 25,000 Stacked-MNIST/CIFAR-10 images, or 50,000 ImageNet/Places365 images.For k-means++, ten clusterings are tried and the one with the best clustering objective is selected.Source: Appendix A · p. 11
  • FID, FSD, and Inception Score use 50,000 fully trained-model samples; FID and FSD are computed against the training set without truncation sampling.Source: Appendix A · p. 11
  • Reverse-KL estimates use D_KL(p_fake || p_real), with distributions formed from empirical classifier predictions and 0 log 0 treated as 0.Source: Appendix A · p. 11
  • Online clustering performs well on 2D-grid and Stacked-MNIST, but not convincingly on real images.The variant updates clusters every training iteration after an initial 25,000 fixed-clustering iterations.Source: Appendix A · p. 11
  • Logo-GAN labels come from clustering pretrained ResNet-50 classifier outputs or an autoencoder latent space, using 50,000 outputs and k = 100.The autoencoder uses a 512-dimensional latent space and is trained with mean-squared loss for 200,000 iterations.Source: Appendix A · p. 11

B.1. Comparison with MGAN

On the 2D-grid benchmark, self-conditioned GANs are more stable than MGAN as cluster count grows; the appendix also reports improved FID, FSD, and IS on combined Places365 and ImageNet data.

B.1. Comparison with MGAN

  • When k exceeds the number of modes, our method levels off with diverse, high-quality samples; when k is smaller, results worsen.The authors therefore favor a larger cluster count to ensure mode coverage.Source: §B.1 · p. 11
  • Our method is more stable and scales well with k on 2D-grid, while MGAN performance degrades as k increases.MGAN's auxiliary classification loss encourages generator outputs to be distinguishable, which the authors hypothesize hinders generators from collaboratively covering a mode; self-conditioned GANs dynamically update cluster weights without requiring distinct conditioned distributions.Source: §B.1 · p. 11, Figure 6 · p. 12
  • With k = 30 for 25 Gaussians, MGAN misses a mode at variance σ^2 = 0.0025 but performs well at σ^2 = 0.1.The cited comparison attributes the missed mode to generators' difficulty working together on a single mode.Source: §B.1 · pp. 11–12, Figure 7 · p. 12

C.1. Hyperparameter Details

  • Synthetic-data experiments use 2-dimensional latents, 400 epochs, batch size 100, and learning rate 10^-3, with 32-dimensional generator conditioning embeddings.Source: §C.1 · p. 12
  • CIFAR-10 and Stacked-MNIST use 128-dimensional latents, batch size 64, learning rate 10^-4, and 256-dimensional generator embeddings.Training lasts 400 epochs for CIFAR-10 and 50 epochs for Stacked-MNIST.Source: §C.1 · p. 12
  • Places365 and ImageNet networks are trained from scratch for 200,000 iterations with batch size 128, learning rate 10^-4, and 256-dimensional generator embeddings.Source: §C.1 · p. 12
  • Our method improves FID, FSD, and IS over vanilla unconditional and class-conditional GANs on combined Places365 and ImageNet datasets.Source: Table 7 · p. 12

C.2. Combined ImageNet and Places365 Experiments

On the combined ImageNet and Places365 dataset, the method outperforms class-conditioned GANs in FID and FSD. The reconstruction appendix describes how its encoder is trained and used to invert images.

C.2. Combined ImageNet and Places365 Experiments

  • On the combined ImageNet and Places365 dataset, the method outperforms class-conditioned GANs in FID and FSD.The experiments use k = 100; the unconditional GAN uses γ = 100 for convergence.Source: §C.2 · p. 12
  • The combined dataset has roughly 3 million images across 1365 classes, with Places365 classes containing more images than ImageNet classes.Class labels are sampled according to their distribution, weighted toward Places365 rather than uniformly.Source: §C.2 · p. 13

C.3. Reconstruction of Real Images

  • The encoder is initialized by separately inverting the generator’s last seven layers, then jointly using those layerwise encoders.The procedure follows GAN-seeing and uses an architecture similar to GANstability’s ResNet2 discriminator.Source: §C.3 · p. 13
  • For generated images, the encoder predicts a latent vector and class embedding; its loss adds cluster classification to GAN-seeing’s objective and uses a modified pixel reconstruction loss.The classification loss is cross-entropy between logits from the predicted embedding and the true cluster index; pixel reconstruction uses the true embedding and predicted latent vector, which improves inversion quality.Source: §C.3 · p. 13
  • To optimize an image’s latent code, the method infers its class from the largest latent-prediction logit and uses the encoder’s seventh-layer output for feature-space loss.Source: §C.3 · p. 13

C.4. Visualizing Generators with GAN Dissection

GAN Dissection finds that the self-conditioned generator has more interpretable units, on par with class-conditional models. Across tested conditions, more of its units change concepts than in the class-conditioned generator.

C.4. Visualizing Generators with GAN Dissection

  • The self-conditioned generator has more interpretable units, on par with class-conditional models, counting units with IoU greater than 0.04.The analysis measures units in early-to-mid generator layers.Source: §C.4 · p. 13, Figure 8 · p. 13
  • The analysis examines how a generator unit’s concept changes with the conditioned cluster, treating IoU above 0.04 as interpretable.Source: §C.4 · p. 13
  • Across 20 conditions, 663 self-conditioned GAN units change meaning, compared with 562 class-conditioned GAN units in a 1024-unit layer.Some units change concepts across clusters, while others are reused for the same concepts, such as water and sky.Source: §C.4 · p. 13

C.5. Interaction with discriminator regularization

With regularization turned off and a smaller architecture, the method matches supervised models’ ImageNet and Places365 performance. The authors report that stronger regularization hurts their method and associate it with worse discriminator features for clustering.

C.5. Interaction with discriminator regularization

  • The method performs best with weak regularization, and its performance suffers when regularization is too strong.Source: §C.5 · p. 13
  • With regularization off and half as many layers as the adapted ResNet-based generator, the method achieves FID 22.68 on Places365 and FID 55.26 on ImageNet.Class-conditional scores are FID 29.33 on Places365 and FID 53.36 on ImageNet; unconditional scores are FID 75.09 and FID 87.24, respectively.Source: §C.5 · p. 14
  • The authors hypothesize that regularization weakens discriminator features for clustering; a linear classifier on R1-regularized features reaches 30.7% top-1 accuracy versus 59.8% on unregularized features.The compared discriminators are fully class-conditional.Source: §C.5 · p. 14

Appendix D. Additional Qualitative Results

Qualitative comparisons on Places365 and ImageNet show wider sample diversity for the self-conditioned GAN than for competing methods. Additional figures display inferred clusters and their generated samples.

Appendix D. Additional Qualitative Results

  • Additional visual results show inferred clusters and corresponding generated samples for Places365 and ImageNet.Source: Appendix D · p. 14, Figure 12 · p. 17
  • On Places365, the self-conditioned GAN captures wider diversity than competing methods, which often show severe mode collapse and similar patterns.Samples are ranked by a classifier score for a single category.Source: Appendix D · p. 14, Figure 10 · p. 15
  • On ImageNet, the self-conditioned GAN captures wider diversity than competing methods, which often show severe mode collapse and similar patterns.Samples are ranked by a classifier score for a single category.Source: Appendix D · p. 14, Figure 11 · p. 16

Related papers

Loading 2006.10728v2…