PaperlayerView exact revision on arXiv

Source-linked AI summary

Instance-Conditioned GAN

Arantxa Casanova, Marlène Careil, Jakob Verbeek, Michal Drozdzal, Adriana Romero-Soriano

arXiv:2109.05070v2cs.CVcs.LG

TL;DR

Complex datasets remain difficult for unconditional GANs because their distributions are hard to model effectively. IC-GAN addresses this by learning overlapping neighborhood distributions around conditioning instances, improving generation across datasets and supporting controllable class-conditional synthesis. It reports gains over unconditional and unsupervised-partitioning baselines, competitive ImageNet results, and improvement over BigGAN on ImageNet-LT.

  • Problem

    Unconditional GANs struggle to model complex distributions such as ImageNet and COCO-Stuff, while unsupervised partitioning baselines remain below class-conditional quality.

  • Method

    IC-GAN conditions its generator and discriminator on instance features and trains them using overlapping nearest-neighbor neighborhoods as local data distributions.

  • Results

    IC-GAN improves unlabeled generation on ImageNet and COCO-Stuff, transfers to unseen datasets by changing conditioning instances, and achieves competitive class-conditional results while surpassing BigGAN on ImageNet-LT.

  • Takeaways & Limitations

    Instance conditioning supports local-density modeling, dataset transfer, and semantically controllable generation within the reported unlabeled and class-conditional settings.

  • Takeaways & Limitations

    IC-GAN requires storing training instances and depends on a pretrained feature extractor for its conditioning representations.

Abstract

from arXiv · show

Generative Adversarial Networks (GANs) can generate near photo realistic images in narrow domains such as human faces. Yet, modeling complex distributions of datasets such as ImageNet and COCO-Stuff remains challenging in unconditional settings. In this paper, we take inspiration from kernel density estimation techniques and introduce a non-parametric approach to modeling distributions of complex datasets. We partition the data manifold into a mixture of overlapping neighborhoods described by a datapoint and its nearest neighbors, and introduce a model, called instance-conditioned GAN (IC-GAN), which learns the distribution around each datapoint. Experimental results on ImageNet and COCO-Stuff show that IC-GAN significantly improves over unconditional models and unsupervised data partitioning baselines. Moreover, we show that IC-GAN can effortlessly transfer to datasets not seen during training by simply changing the conditioning instances, and still generate realistic images. Finally, we extend IC-GAN to the class-conditional case and show semantically controllable generation and competitive quantitative results on ImageNet; while improving over BigGAN on ImageNet-LT. Code and trained models to reproduce the reported results are available at https://github.com/facebookresearch/ic_gan.

1 Introduction

IC-GAN addresses the difficulty of modeling complex image distributions by learning overlapping local densities around individual instances. It improves unlabeled generation, supports transfer to unseen datasets, and extends to controllable class-conditional generation.

  • Motivation: Unconditional GANs struggle with mode collapse, limited distribution coverage, and complex datasets such as ImageNet.Class-conditional GANs ease learning by partitioning data with labels.
  • Motivation: Unsupervised partitioning methods use coarse, non-overlapping clusters that may mix object or scene types, degrading generated-sample quality.Finer partitions can leave too few samples for reliable generator and discriminator training.
  • Approach: IC-GAN models each instance’s neighborhood by conditioning the generator and discriminator on instance representations and using neighboring instances as real samples.Large overlapping neighborhoods avoid excessively small clusters while preserving local density structure.
  • Approach: IC-GAN resembles kernel density estimation by representing the dataset as a mixture of local components, but models those densities implicitly with an adversarial neural network.The model conditions on an instance and noise vector rather than explicitly modeling likelihood.
  • Results: IC-GAN outperforms prior unlabeled-generation approaches on ImageNet and COCO-Stuff, transfers to unseen datasets by changing conditioning instances, and achieves competitive class-conditional results.The class-conditional extension provides controllable semantics through class and instance conditioning.

2 Instance-conditioned GAN

IC-GAN represents data with overlapping nearest-neighbor neighborhoods and conditions both generator and discriminator on instance features. The generator learns to produce samples resembling an instance’s neighborhood, with class labels added for class-conditional generation.

  • Neighborhood modeling: IC-GAN approximates the data distribution as a mixture of conditional distributions around instance feature vectors.Each local cluster is described by a datapoint and its nearest neighbors in feature space.
  • Neighborhood modeling: Instance features are extracted with an embedding function, and each datapoint’s neighborhood consists of its k nearest neighbors under cosine similarity.These neighborhoods can contain datapoints from different classes.
  • Adversarial training: The generator maps Gaussian noise and an instance feature hi to a synthetic sample, while the discriminator distinguishes generated samples from real neighbors conditioned on the same hi.Real neighbors are sampled uniformly from the instance’s nearest-neighbor set.
  • Inference: At inference, IC-GAN requires instance features that may come from either the training distribution or a different dataset.This enables conditioning on instances outside the training distribution.
  • Class-conditional extension: For class-conditional generation, both generator and discriminator additionally receive a class label, while neighborhoods remain defined by feature similarity.Nearest-neighbor sets may include fewer than k examples from the instance’s class.

3 Experimental evaluation

The experiments evaluate IC-GAN on unlabeled ImageNet and COCO-Stuff, including dataset transfer, class-conditional ImageNet-LT generation, and sensitivity to stored instances and neighborhood size. Across these settings, IC-GAN improves generation quality or diversity relative to relevant baselines, while larger instance coverage trades slightly lower precision for better recall.

  • Unlabeled ImageNet: IC-GAN surpasses previous unlabeled ImageNet approaches in FID and IS at 64×64 and 128×128, and outperforms the reported unconditional diffusion model at 256×256.The comparison uses an unconditional BigGAN baseline and other prior approaches; Table 1 includes non-official PyTorch FID and IS values for one baseline comparison.
  • COCO-Stuff: On COCO-Stuff, StyleGAN2-based IC-GAN achieves state-of-the-art FID scores, including against bounding-box-conditioned LostGANv2 and OC-GAN.IC-GAN matches or improves the same-backbone unconditional model in all reported cases except StyleGAN2 at 256×256 training FID, while using architectures with fewer parameters than the tailored baselines.
  • Dataset transfer: Changing conditioning instances implements a distribution shift: replacing COCO-Stuff instances with ImageNet instances yields a 43.5 train FID.The transferred model also outperforms LostGANv2 and OC-GAN, while larger ImageNet training neighborhoods may contribute to higher diversity.
  • Dataset transfer: ImageNet-trained IC-GAN preserves the semantics and style of Cityscapes, MetFaces, and PACS images, although quality degrades for datasets more unlike ImageNet.These transfer experiments change the conditioning dataset without retraining the model.
  • Class-conditional generation: On ImageNet-LT, IC-GAN improves FID and IS over BigGAN across many-shot, medium-shot, and few-shot class groups.The evaluation samples from all 115k available training instances to avoid ignoring rare classes under the dataset’s imbalance.
  • Ablations: Storing more instances improves recall but slightly worsens precision, while k-means selection outperforms random selection and larger neighborhoods help when few instances are stored.With k-means, 1,000 stored instances outperform 5,000 randomly selected instances, and storing more than 1,000 gives no noticeable FID improvement.

4 Related work

Prior work partitions data or conditions generation using feature vectors, but often uses coarse clusters, reconstruction objectives, or complex labeled pipelines. IC-GAN instead models localized distributions around individual data points with a single conditional generator, extending nonparametric density modeling to complex and long-tailed settings.

  • GAN data partitioning: Existing GAN approaches mitigate mode collapse through clustering, mixture models, or discriminator designs using multiple samples.These methods aim to improve image-generation quality and diversity by partitioning the data manifold or modifying adversarial training.
  • GAN data partitioning: Cluster-conditioned methods use relatively coarse partitions that can mix object or scene types, while finer clusters may lack enough samples to model their distributions.This creates a trade-off between heterogeneous low-density regions and undersized clusters.
  • Feature conditioning: Feature-conditioned generative modeling has also used self-supervised representations for reconstruction and required storing features for all training samples.IC-GAN instead learns a localized distribution around nearest-neighbor images and stores only a small feature subset.
  • Nonparametric density modeling: IC-GAN models dataset density nonparametrically around each data point with a single conditional generation network, unlike prior localized adversarial-density approaches.Its design connects adversarial training with kernel-density-style local modeling without explicitly modeling likelihood.
  • Complex scene generation: Unlike labeled complex-scene pipelines, IC-GAN uses instance conditionings for global semantic control without dataset labels and addresses long-tailed generation settings.The related-work discussion identifies long-tail class distributions as a challenge for class-conditional GANs.

5 Discussion

The paper reports broad generation and transfer capabilities across labeled and unlabeled settings, while identifying storage, feature-extractor dependence, and transfer distance as limitations. IC-GAN also supports controllable changes in style or semantics through its conditioning variables.

  • Discussion: IC-GAN reports consistent improvements over baselines on unlabeled ImageNet and COCO-Stuff and competitive ImageNet class-conditional results.It also surpasses BigGAN on ImageNet-LT.
  • Discussion: The model can produce samples from different data distributions by conditioning an ImageNet-trained model on instances from other datasets.The paper describes these transfer samples as compelling.
  • Discussion: Class-conditional IC-GAN enables controllable generation by changing either the instance or the class-conditioning while holding the other conditioning fixed.Changing the instance controls style, while changing class conditioning controls semantics.
  • Limitations: IC-GAN requires storing training instances, depends on a pretrained feature extractor, and degrades in quality when transferred to datasets very different from ImageNet.The paper notes that approximately 1,000 instances can cover ImageNet and proposes joint feature-extractor and generator training as future work.

Instance-Conditioned GAN: Supplementary Material

The supplementary material documents assets, experimental resources, architecture extensions, and instance-feature processing for IC-GAN experiments. It also describes replacing class embeddings with learned layers that incorporate instance features into BigGAN and StyleGAN2 backbones.

  • Supplementary overview: The supplementary material includes asset links and licensing information, experimental setup, hyperparameters, metrics, and additional qualitative and transfer results.These materials are organized across the supplementary sections and tables.
  • Architecture modifications: IC-GAN experiments use BigGAN and StyleGAN2 backbones extended to accept instance conditionings.The extensions adapt the original class-conditional architectures for instance-based inputs.
  • Architecture modifications: With BigGAN, class embedding layers are replaced by fully connected layers that process 2,048-dimensional instance features in the generator and discriminator.The generator output dimensionality is set to 512 in the reported experiments.
  • Architecture modifications: With StyleGAN2, instance features are mapped through a 512-dimensional generator layer and concatenated with noise at the mapping-network input.This creates a style vector conditioned on the instance.
  • Instance features: All instance feature vectors are L2-normalized before neighborhood computation and before serving as GAN conditioning inputs.Normalization is applied both during neighborhood construction and model use.

B.2 Training details and hyperparameters

Training uses FID-based stopping and dataset-specific hyperparameter searches across BigGAN and StyleGAN2 configurations. The experiments also vary augmentation, compute resources, and update schedules by dataset and resolution.

  • Training procedure: Training stops after FID fails to improve for 50 epochs or when training FID diverges.The same monitoring framework is used across the reported models, with iteration equivalents adjusted for batch size.
  • ImageNet: ImageNet BigGAN experiments use dataset- and resolution-specific settings, with IC-GAN and BigGAN generally using one discriminator update in the reported 64×64 configuration.Additional batch-size and channel-width configurations did not improve the selected setup.
  • COCO-Stuff: COCO-Stuff BigGAN selection identifies batch size 256 and channel width 48 as the best configuration among the explored choices.Learning rates and discriminator-update counts were also searched across several candidate values.
  • ImageNet-LT: ImageNet-LT selection identifies batch size 128 and channel width 64 as the best configuration, with learning rates and update counts varying by resolution.The reported 64×64 setup uses one discriminator update for both BigGAN and IC-GAN.
  • Augmentation and resources: Horizontal flips augment real data in all experiments, while DiffAugment translations improve FID on COCO-Stuff and ImageNet-LT but not ImageNet.Training uses between one and 32 NVIDIA V100 GPUs depending on dataset and resolution.

C Additional metrics: Precision and Recall

Precision and Recall provide complementary measures of IC-GAN’s visual quality and diversity, with results favoring IC-GAN on several ImageNet settings.

  • IC-GAN achieves better Recall than all baselines in both unlabeled and labeled ImageNet settings when selecting 10,000 random training instances.Recall is interpreted as a measure of diversity.
  • With 1,000 k-means-selected instances, IC-GAN obtains higher Precision than other baselines in the unlabeled setting.Precision is used as a measure of visual quality.
  • In the labeled setting, IC-GAN’s Precision exceeds BigGAN at 64×64 but is lower at 128×128 and 256×256.

D Comparison between StyleGAN2 and BigGAN backbones on ImageNet

IC-GAN improves StyleGAN2’s ImageNet metrics across resolutions, while backbone behavior differs between ImageNet and COCO-Stuff.

  • IC-GAN with a StyleGAN2 backbone obtains better FID and IS than StyleGAN2 across all ImageNet resolutions.This supports that IC-GAN does not depend on a specific backbone.
  • StyleGAN2 is outperformed by unconditional BigGAN on ImageNet, whereas StyleGAN2 improves over BigGAN on the smaller COCO-Stuff dataset.The authors relate this pattern to differences in dataset complexity and architecture design context.
  • On COCO-Stuff, people and faces were qualitatively better generated with a StyleGAN2 backbone than with BigGAN.

E Additional qualitative results for IC-GAN

Additional qualitative results show that IC-GAN preserves conditioning semantics, supports smooth instance-feature control, and generates diverse scenes and class-conditional images.

  • Unlabeled ImageNet: IC-GAN generations generally preserve the semantics and style of ImageNet conditioning instances while resembling nearby real samples.Examples include animals, landscapes, and water-related scene content.
  • Instance feature vector and noise interpolation: Interpolating instance features changes generated attributes such as fur color, camera proximity, color, appearance, and mixtures of tiger stripes with wolf fur.Noise interpolation is varied horizontally while instance-feature interpolation is varied vertically.
  • Unlabeled COCO-Stuff: On COCO-Stuff, IC-GAN generates higher-quality and more diverse scenes than LostGANv2 and OC-GAN across the qualitative comparisons.The competing methods use labeled bounding-box layouts, whereas IC-GAN conditions on features extracted from real samples.
  • ImageNet: Class-conditional IC-GAN produces ImageNet images with quality similar to BigGAN in shown cases while enabling semantic control through instance features.Changing conditioning instances alters backgrounds, environments, and viewpoints within classes such as goldfish, limousines, and red foxes.
  • Swapping classes for class-conditional IC-GAN: Using instance features from other classes can preserve aspects of the conditioning context while replacing the requested object with another class.Examples include generating camels or zebras in a snowy setting from snowplow features.

F Additional off-the-shelf transfer results for IC-GAN

Off-the-shelf transfer experiments show that changing conditioning instances shifts IC-GAN’s generated distribution, while transferred outputs retain conditioning-related features and training-dataset style.

  • Transfer across datasets: Conditioning an ImageNet-trained IC-GAN on COCO-Stuff instances yields better COCO-Stuff generation than conditioning it on ImageNet instances.The reported FID values are 8.5 for COCO-Stuff instances against COCO-Stuff and 43.6 for ImageNet instances against COCO-Stuff.
  • Transfer across datasets: Changing instance features shifts generated images toward the reference dataset, with COCO-Stuff-conditioned features producing 37.2 FID against ImageNet.The experiment uses 1,000 k-means-selected instance features and 76,000 generated samples.
  • What is transferred: The transferred component is a function predicting kernel shape from the conditioning instance, rather than kernel location.The generator probes this predicted local density by sampling from the model.
  • Transfer behavior: Transferred conditioning can retain scene characteristics while adapting object appearance and color palette to the training dataset’s style.Examples include giraffe-like animals without characteristic giraffe features and colorful outputs from Cityscapes instances.
  • Unlabeled transfer results: Unseen instances from COCO-Stuff, Cityscapes, MetFaces, PACS, and Sketch can produce outputs that differ substantially from their closest ImageNet training images.Sketch-conditioned examples can resemble sketch strokes even when the closest ImageNet samples are nonsketch objects.
  • Class-conditional transfer: Class-conditional transfer combines ImageNet class labels with unseen instances from other datasets to generate class-specific objects in transferred contexts.Reported examples include camels in grass from a cow instance and zebras in urban settings from a Cityscapes instance.
  • ImageNet-LT: Balancing ImageNet-LT class distributions harmed performance in nearly all evaluated cases except validation Inception Score.The authors hypothesize that oversampling rare classes may overfit the discriminator.
  • Feature extractors: IC-GAN performance is similar with self-supervised SwAV and supervised RN50 feature extractors, suggesting limited sensitivity to feature-extractor choice.

J Matching storage requirements for IC-GAN and unconditional models

The authors test whether IC-GAN’s gains remain after matching unconditional baselines for parameter capacity and instance-feature storage. They expand unconditional BigGAN and StyleGAN2 models to compensate for IC-GAN’s additional requirements.

  • The comparison tests whether IC-GAN’s performance can be attributed solely to extra parameters and memory, rather than its design.The design includes finegrained overlapping partitions and instance conditionings.
  • The ImageNet setup resembles the dataset’s class partition, where 1,000 classes contain approximately 1,200 images each.
  • Unconditional BigGAN models are trained with all labels set to zero and increased capacity to match IC-GAN’s storage requirements.The capacity compensation covers both additional model parameters and the instances required by IC-GAN.
  • ImageNet IC-GAN adds 4.5M parameters and stores 1,000 instance features requiring 8MB, roughly equivalent to 2M parameters.The unconditional BigGAN capacity is increased by expanding both generator and discriminator width.
  • COCO-Stuff IC-GAN requires 4M additional parameters plus 8MB for 1,000 instance features, while StyleGAN2 instance conditionings add 1M parameters.The corresponding unconditional baselines are enlarged to compensate for these requirements.

K Additional neighborhood size impact studies

The additional studies show that neighborhood size k affects both quantitative performance and qualitative generation. Smaller neighborhoods favor quality with less diversity, whereas larger neighborhoods increase variation but can reduce quality.

  • The stored parameters and instance features are represented as float32.
  • k = 5 achieves the best FID and IS metrics for IC-GAN with a BigGAN backbone on both ImageNet-LT and COCO-Stuff.The studies report this result for the two smaller datasets, compared with k = 50 in the ImageNet case.
  • Increasing k in COCO-Stuff and ImageNet-LT could include semantically different samples because these datasets contain fewer semantically similar neighbors.The passage presents this as a potential explanation for the lower preferred neighborhood size.
  • Smaller neighborhoods produce less diverse images, while k = 500 produces more varied but lower-quality generations.The qualitative results support the interpretation that k controls the smoothing effect.
  • Table 16 uses class-conditional IC-GAN on ImageNet-LT, while Table 17 evaluates IC-GAN on COCO-Stuff using the stated dataset-specific feature extractors.ImageNet-LT uses a classifier-trained ResNet50, whereas COCO-Stuff uses a self-supervised SwAV ResNet50.
Loading 2109.05070v2…