Source-linked AI summary
High-Fidelity Image Generation With Fewer Labels
Mario Lucic, Michael Tschannen, Marvin Ritter, Xiaohua Zhai, Olivier Bachem, Sylvain Gelly
TL;DR
High-fidelity image generation has depended on vast labeled datasets, despite the cost and scarcity of labels and a quality gap between conditional and unsupervised models. This paper combines self-supervised feature learning with semi-supervised label inference for GAN training. It achieves state-of-the-art unsupervised ImageNet generation, matches BIGGAN with 10% of labels, and outperforms it with 20%.
Problem
High-fidelity natural image generation depends on vast labeled datasets, while labeling is costly and the quality gap between conditional and unsupervised generation remains significant.
Method
The approach uses self-supervised semantic representations and semi-supervised inference of labels from a small labeled subset to guide conditional GAN training.
Results
The method achieves state-of-the-art unsupervised ImageNet generation, matches BIGGAN with 10% of labels, and sets a new FID state of the art with 20%.
Takeaways & Limitations
Self- and semi-supervised learning can reduce the labeled-data requirement for state-of-the-art natural image synthesis in both unsupervised and supervised settings.
Takeaways & Limitations
Further progress may be hindered by engineering challenges involved in training large-scale generative adversarial networks.
Abstract
from arXiv · showhide
Deep generative models are becoming a cornerstone of modern machine learning. Recent work on conditional generative adversarial networks has shown that learning complex, high-dimensional distributions over natural images is within reach. While the latest models are able to generate high-fidelity, diverse natural images at high resolution, they rely on a vast quantity of labeled data. In this work we demonstrate how one can benefit from recent work on self- and semi-supervised learning to outperform the state of the art on both unsupervised ImageNet synthesis, as well as in the conditional setting. In particular, the proposed approach is able to match the sample quality (as measured by FID) of the current state-of-the-art conditional model BigGAN on ImageNet using only 10% of the labels and outperform it using 20% of the labels.
1. Introduction
High-fidelity image generation currently depends on abundant labeled data, motivating methods that reduce this dependence. The paper combines self-supervised representations with semi-supervised label inference to improve label efficiency while matching or exceeding supervised-generation performance.
- High-fidelity natural image generation hinges on vast quantities of labeled data, although most data is unlabeled and labeling is costly and error-prone.
- The paper targets the significant sample-quality gap between conditional and unsupervised image generation.
- The approach learns a semantic feature extractor through self-supervision and uses its representation to guide GAN training.
- Labels for the full training set can be inferred from a small labeled subset and used as conditional information for GAN training.
- S3GAN matches state-of-the-art conditional generation with 10% of ImageNet labels and outperforms it with 20%.
2. Background and related work
Prior ImageNet GANs largely rely on class labels, while related semi-supervised and self-supervised methods provide alternatives for learning from less labeled data. The paper situates its approach among these label-dependent, semi-supervised, and representation-learning strategies.
- Most prior GANs scaled to ImageNet rely on class-conditional generation using labels, including auxiliary-classifier and projection-based approaches.
- Semi-supervised GAN methods train discriminators or classifiers to distinguish image classes while handling generated samples separately or through extended objectives.
- Self-supervised learning uses label-free auxiliary tasks to learn semantic data representations across modalities, including images.
- S3GAN matches BIGGAN in FID and IS using only 10% of ground-truth labels at 128×128 resolution.
3. Reducing the appetite for labeled data
The paper reduces reliance on ground-truth labels by replacing them with labels inferred through self-supervised, semi-supervised, or co-training approaches. These methods range from pre-training representations and classifiers before GAN training to inferring labels within the GAN itself.
- Overview: The proposed approaches replace hand-annotated labels for real images with inferred labels to improve generative modeling with few labels.The paper focuses on using few labels to improve the generative model rather than training a classifier from few labels.
- Conditional GAN setup: Projection-discriminator GANs combine an unconditional real/fake classifier with a class-conditional projection layer, while the generator incorporates labels through conditional BatchNorm.The discriminator is represented as D(x, y) = cr/f(˜D(x)) + P(˜D(x), y), with P(˜x, y) = ˜x⊤Wy.
- Pre-trained approaches: CLUSTERING learns a feature extractor from rotation-prediction self-supervision, clusters the resulting training-image representations, and uses cluster assignments as GAN labels.The rotation task uses the four angles 0°, 90°, 180°, and 270°; mini-batch k-Means then produces the inferred labels.
- Pre-trained approaches: S2GAN jointly learns a self-supervised feature representation and a linear classifier from the labeled subset, then predicts labels for real images during GAN training.The self-supervised loss uses all training images, while the cross-entropy term uses the labeled subset; γ balances the two terms.
- Co-training approach: Co-training avoids separately pre-training a feature extractor and label-inference mechanism by learning an auxiliary classifier on discriminator features during GAN training.S2GAN-CO trains the auxiliary classifier on labeled real examples and uses it to predict labels for unlabeled real images.
- Unsupervised and co-training variants: Unsupervised variants either remove class conditioning with a single label or assign random labels, while rotation-prediction can provide an additional self-supervised task during GAN training.The single-label variant removes the discriminator projection layer; random labels can add randomness and trainable class-conditional BatchNorm parameters.
4. Experimental setup
The experiments compare unsupervised and semi-supervised variants of BIGGAN on 128 × 128 ImageNet, using established image-generation metrics and repeated runs. The setup also specifies the architecture, optimization choices, datasets, and self-/semi-supervised configurations.
- Architecture and training: The implementation uses the conditional BIGGAN architecture with Adam, spectral normalization, 250k generator steps, and synchronized BatchNorm across 128 TPU cores.The architecture matches released BIGGAN checkpoints at byte level, and the code is open-sourced for reproducibility.
- Datasets: ImageNet provides 1.3M training images and 50k test images across 1k object classes, resized to 128 × 128 × 3.Semi-supervised datasets randomly select k% of samples from each class for labeling.
- Evaluation: FID and Inception Score evaluate generated-sample quality using Inception-network embeddings and label-distribution statistics.FID compares empirical means and covariances of real and generated samples, while Inception Score measures conditional-label confidence and sample variability.
- Methods: The study compares unmodified BIGGAN with unsupervised SINGLE LABEL, RANDOM LABEL, and CLUSTERING, plus semi-supervised S2GAN and S2GAN-CO.S2GAN-CO uses soft labels, while S2GAN uses hard labels by default and separately investigates soft labels.
- Evaluation: Every model is trained three times with different random seeds, and the median FID and median Inception Score are reported.The reported metrics average results across five randomly sampled fake-image sets.
- Method configurations: CLUSTERING selects the number of clusters from {50, 100, 200, 500, 1000}, while self-supervised GAN variants use α = 0.2 and selected β values.The pre-trained feature extractor uses a wide ResNet-50 v2, and the semi-supervised classifier configuration uses γ = 0.5 and B = 1536.
5. Results and discussion
The experiments show that self- and semi-supervised approaches substantially reduce the dependence of high-fidelity ImageNet GAN synthesis on ground-truth labels, although fully unsupervised models remain behind fully supervised BIGGAN. Self-supervision during GAN training consistently improves results, enabling parity with BIGGAN at 10% labels and better performance at 20%.
- BIGGAN reproduces the reported supervised baseline with FID 8.4 and IS 75.0.
- Unsupervised approaches: Fully unsupervised RANDOM LABEL and SINGLE LABEL models achieve similar FID of ∼25 and IS of ∼20, leaving a considerable gap to BIGGAN.One SINGLE LABEL run collapsed, while all RANDOM LABEL runs trained stably for 250k generator iterations.
- Unsupervised approaches: CLUSTERING reduces FID by about 10% and increases IS by about 10%, establishing a new state of the art in unsupervised ImageNet generation.The result used 50 clusters; performance remained considerably worse than BIGGAN.
- Semi-supervised approaches: S2GAN matches BIGGAN with 20% labels and incurs only a minor FID increase with 10% and 5% labels despite imperfect inferred-label accuracy.The classifier top-1 accuracies are 50%, 63%, and 71% for 5%, 10%, and 20% labeled data, respectively.
- Semi-supervised approaches: S2GAN-CO outperforms fully unsupervised approaches across all label percentages, reaching FID 13.9 and IS 49.2 with 20% labels.At 20% labels, these values are reported as remarkably close to BIGGAN and S2GAN; performance gaps increase as labels decrease.
- Self-supervision during GAN training: Self-supervision during GAN training reduces FID and increases IS across settings, matching BIGGAN with 10% labels and outperforming it with 20%.For S3GAN, FID improvements are around 10% in most cases, while other approaches improve by 5 to 10%.
6. Conclusion and future Work
The paper shows that self- and semi-supervised learning can reduce the need for labeled data while achieving state-of-the-art natural image synthesis. It frames these results as progress toward few-shot high-fidelity image generation, with several directions remaining for future work.
- Self- and semi-supervised learning enabled state-of-the-art results in both unsupervised and supervised natural image synthesis.
- The authors identify few-shot high-fidelity image synthesis as the broader goal that these results begin to address.
- Future work includes testing larger and more diverse datasets, other self- and semi-supervised methods, and other deep generative models.
A. Additional samples and interpolations
The additional samples illustrate latent-space interpolation, cluster-level realism and diversity, class-conditional generation, and remaining challenges involving human faces and dynamic scenes.
- Latent-space interpolations are shown for S3GAN trained with 20% labels at 128 × 128 and 10% labels at 256 × 256.
- Real and generated samples from clustered data include underwater scenes, animal-focused outdoor scenes, and diverse indoor and outdoor scenes.
- At 20% labels and 128 × 128 resolution, S3GAN captures within-class diversity, while human faces and dynamic scenes remain challenging.
- At 20% labels and 128 × 128 resolution, generated samples correctly distinguish classes without observed class leakage.
- At 10% labels and 256 × 256 resolution, generated samples capture diversity within a single class.
B. Architectural details
The appendix specifies the ResNet generator and discriminator architectures, their tensor-level parameters, and the notation used for resampling, normalization, labels, and feature representations.
- The ResNet implementation follows Brock et al. and uses RS, BN, and cBN to denote resampling, batch normalization, and conditional batch normalization.
- The generator and discriminator use ResBlocks whose shortcut and convolution paths are merged by addition, with dimensions specified by input height, width, and channel counts.
- The discriminator applies spectral normalization to all layers, while the architecture tables specify channel width and resampling settings.
- In the generator notation, y denotes labels and h denotes the pre-logit layer output.
- The generator contains 70,433,988 trainable parameters, and the discriminator contains 87,982,370.
C. FID and IS training curves
The training-curve figure reports mean FID and IS across three ImageNet runs as generator steps increase, with stable training for all models except SINGLE LABEL in one run.
- Mean FID and IS are evaluated over three ImageNet runs as a function of generator steps.The curves use 128 × 128 ImageNet models.
- All models train stably except SINGLE LABEL, for which one run collapsed.
D. FID and IS: Mean and standard deviations
The tables compare pre-trained and co-training strategies, assess self-supervision during GAN training, and examine hard versus soft predicted labels for unsupervised approaches.
- Pre-trained approaches outperform co-training approaches, which in turn outperform fully unsupervised approaches.
- Self-supervision during GAN training improves results across all evaluated approaches.
- Training with hard predicted labels produces better models than training with soft predicted labels.
- The unsupervised approaches are evaluated using mean FID and IS.