Source-linked AI summary
Large Scale Adversarial Representation Learning
Jeff Donahue, Karen Simonyan
TL;DR
The paper asks whether generative models can provide strong unsupervised representations despite self-supervision’s dominance. It extends BigGAN into BigBiGAN with an encoder and modified discriminator, and reports state-of-the-art ImageNet representation learning and unconditional generation. The results support continued progress through stronger generative and inference models, while the reported evidence includes specific architectural and training-scope caveats.
Problem
Generative models had shown limited realization of representation learning from raw data, raising how their learned semantics could be leveraged for representations.
Method
BigBiGAN extends BigGAN with an encoder, a joint discriminator, additional unary loss terms, and a more stable discriminator design.
Results
BigBiGAN matches state-of-the-art unsupervised ImageNet representation learning and achieves state-of-the-art unconditional ImageNet generation.
Takeaways & Limitations
The findings support the paper’s conclusion that advances in generative and inference models can benefit unsupervised representation learning and large-scale generation.
Takeaways & Limitations
An alternative discriminator loss applying the hinge once to the summed terms performed significantly worse than separately clamping the three terms.
Abstract
from arXiv · showhide
Adversarially trained generative models (GANs) have recently achieved compelling image synthesis results. But despite early successes in using GANs for unsupervised representation learning, they have since been superseded by approaches based on self-supervision. In this work we show that progress in image generation quality translates to substantially improved representation learning performance. Our approach, BigBiGAN, builds upon the state-of-the-art BigGAN model, extending it to representation learning by adding an encoder and modifying the discriminator. We extensively evaluate the representation learning and generation capabilities of these BigBiGAN models, demonstrating that these generation-based models achieve the state of the art in unsupervised representation learning on ImageNet, as well as in unconditional image generation. Pretrained BigBiGAN models -- including image generators and encoders -- are available on TensorFlow Hub (https://tfhub.dev/s?publisher=deepmind&q=bigbigan).
1 Introduction
BigBiGAN revisits generative models for unsupervised representation learning by pairing a high-quality BigGAN generator with an encoder and modified discriminator. The paper argues that stronger generation can support stronger representations and reports state-of-the-art ImageNet representation learning and unconditional generation.
- Motivation: Generative models can capture complex, multimodal, high-resolution image distributions, motivating their use for learning from raw data.The paper contrasts this with self-supervised methods that modify or withhold parts of the input.
- Prior generative representation learning: BiGAN and ALI add an encoder that maps real data to latents, enabling inference or feature representations alongside generation.Earlier BiGAN work showed effective ImageNet representation learning, but its DCGAN-style generator produced low-quality images and limited modeled semantics.
- Contributions: BigBiGAN combines the BiGAN framework with BigGAN to improve generative modeling and representation learning on ImageNet.The approach extends a high-quality generator-based model rather than relying on self-supervised prediction tasks.
- Results: BigBiGAN matches state-of-the-art unsupervised representation learning on ImageNet and achieves state-of-the-art unconditional ImageNet generation.The paper also reports that the representation-learning objective improves unconditional image generation.
- Contributions: The paper proposes a more stable joint discriminator and analyzes model design choices through extensive empirical evaluation and ablations.The discriminator modification is presented as part of the framework’s representation-learning and generation evaluation.
2 BigBiGAN
BigBiGAN extends BiGAN with BigGAN architectures and a joint discriminator that compares encoder-data pairs with generator-latent pairs. Additional unary data and latent terms guide optimization while preserving the core joint-distribution matching objective.
- Framework: BigBiGAN uses an encoder E for data-to-latent mapping, a generator G for latent-to-data mapping, and a discriminator D over data-latent pairs.The encoder and generator define the two joint distributions that the discriminator distinguishes.
- Framework: The joint discriminator distinguishes pairs from the data distribution and encoder from pairs produced by the generator and latent distribution.The generator and encoder are trained to make these two joint distributions indistinguishable.
- Objective: With an optimal discriminator, the BiGAN objective minimizes Jensen-Shannon divergence between the encoder-data and generator-latent joint distributions.At the global optimum, the two joint distributions match; deterministic encoder and generator functions are then inverses.
- Architecture and loss: BigBiGAN adopts BigGAN generator and discriminator architectures and adds unary terms depending separately on data x and latents z.The unary x term provides a generator signal matching the original GAN objective, while the added terms guide optimization toward matching marginals.
- Architecture and loss: The discriminator loss separately clamps the unary data, unary latent, and joint terms with hinge losses.An alternative that applies the hinge once to their sum performed significantly worse.
- Optimization: Encoder and generator parameters minimize LEG while discriminator parameters minimize LD, with expectations estimated using minibatches.The losses use Monte Carlo estimates over samples from the data and latent distributions.
3 Evaluation
BigBiGAN is evaluated through ablations of its architecture, latent encoder, losses, and resolutions, followed by comparisons on ImageNet representation learning and unconditional generation. Results indicate that generator capacity and higher-resolution encoders improve performance, while BigBiGAN reaches strong classification and generation results.
- Evaluation protocol: BigBiGAN is trained on unlabeled ImageNet, then evaluated with frozen encoder features using supervised linear classification alongside IS and FID generation metrics.The ablation protocol also reports means and standard deviations across three random seeds where possible.
- Model variants: The base model combines BigGAN generator components with an encoder and discriminator architecture whose encoder is stochastic, sampling z from a predicted distribution.The encoder predicts µ and a standard deviation, then uses reparameterized sampling to obtain z.
- Loss ablations: Removing the x unary loss substantially worsens generation, while the z unary term has a smaller effect and can improve FID but worsen IS.The x unary term also marginally improves classification performance.
- Generator capacity: Smaller generators reduce representation and generation quality, confirming that a powerful image generator is important for learning good encoder representations.The one-third-capacity Small G (32) model is unstable, while the two-thirds-capacity Small G (64) model performs modestly worse in classification.
- Resolution: Increasing generator resolution improves representation learning up to 256 × 256, although the 256-resolution model is slower to train and later experiments use 128 × 128 generation.A higher-resolution encoder also significantly improves generation, especially by FID, even when the generator resolution is unchanged.
- Comparison with prior methods: BigBiGAN improves over prior unsupervised representation-learning results on ImageNet, raising top-1 accuracy from 55.4% to 60.8% and reaching 61.3% with BN+CReLU features.The results match a concurrent CPC-based result, while high-resolution encoders also surpass prior unsupervised BigGAN generation under IS and FID.
- Reconstructions: BigBiGAN reconstructions are produced as G(E(x)) and emphasize semantic, high-level details rather than pixel-perfect fidelity.No explicit reconstruction cost is enforced during training, and reconstructions are not computed at training time.
4 Related work
Related work spans self-supervised image tasks, generative models, and reconstruction-oriented approaches. BigBiGAN is distinguished by modeling full-resolution data without requiring input modification.
- Self-supervised methods create labels automatically through tasks such as patch-location prediction, colorization, motion segmentation, rotation prediction, and exemplar matching.
- Contrastive predictive coding predicts which patches occur at other image locations, extending the broader family of automatically supervised image-learning tasks.
- Generative approaches such as BigBiGAN can use full-resolution images without cropping or modification, allowing representations to be applied directly downstream without domain shift.
- Related generative and reconstruction methods include associative compression networks and VQ-VAEs, which use dataset-level compression or discrete encoding with autoregressive decoding.
5 Discussion
The discussion concludes that BigBiGAN achieves state-of-the-art ImageNet representation learning using only generative modeling. It also connects representation learning and generation as mutually beneficial directions for future work.
- BigBiGAN achieves state-of-the-art image representation learning on ImageNet using an unsupervised approach based purely on generative models.
- The ablation study supports the potential for powerful generative models to benefit representation learning and inference models to improve large-scale generation.
- Future directions include benefiting from advances in generative and inference models and scaling to larger image databases.
Appendix A Model and optimization details
The appendix documents BigBiGAN’s optimization, evaluation procedures, architecture comparisons, and supplementary visualizations. It also includes supervised reference results and links to samples and reconstructions.
- Optimization: Training uses Adam with batch size 2048, alternating two discriminator updates with one joint generator-and-encoder update.
- Optimization: BigBiGAN and linear-classification inputs use ResNet-style augmentation with crop sizes of 128 or 256 rather than 224.
- Evaluation: Ablation linear evaluations hold out 10K ImageNet training images for validation, run for 500K steps, and use early stopping based on trainval accuracy.
- Evaluation: Extended experiments train for 1M steps and evaluate on ImageNet’s official 50K-image validation set for comparison with prior work.
- Reference results: Table 4 reports ImageNet validation accuracy for fully supervised end-to-end training of the architectures used in the representation-learning experiments.
- Visualizations: Figure 3 visualizes 1024 first-layer filters from the RevNet ×4 encoder, including qualitative differences between learning-rate variants.
Appendix B Samples and reconstructions
The appendix presents samples, reconstructions, reconstruction-error criteria, and iterated reconstruction behavior for several BigBiGAN variants. Repeated reconstruction initially preserves some input semantics but eventually loses most original content.
- Samples and reconstructions: Samples are selected by best FID against training-set statistics, with IS and FID reported alongside the corresponding images.
- Samples and reconstructions: Reconstructions are selected by lowest relative pixel-wise ℓ1 error, which compares reconstruction error against an independent random-input baseline.
- Samples and reconstructions: The relative reconstruction metric penalizes degenerate outputs such as mean images that may have low absolute error but lack perceptual similarity.
- Iterated reconstruction: Iterated reconstruction repeatedly applies G(E(·)) to an image, starting with R0(x)=x.
- Iterated reconstruction: After dozens or hundreds of iterations, little content from the original input apparently remains intact, despite semantic retention during initial steps.
Appendix C Nearest neighbors
BigBiGAN representations support unsupervised nearest-neighbor classification and retrieve semantically related ImageNet images, though performance remains below supervised linear-classifier results.
- The benchmark uses an 8192D global-average-pooling feature from the best-performing RevNet ×4, ↑E LR model without evaluation-time data augmentation.
- 38% top-1 accuracy is achieved with one neighbor, rising to 43% with 25 neighbors before slightly declining at 50.Across all neighborhood sizes, the normalized ℓ1 distance D1 outperforms D2.
- Nearest neighbors often match query images in object category, pose, and position, indicating high-level attributes in the learned representation.
- Nearest-neighbor predictions require no supervised parameter learning, but results remain far below the paper’s linear-classifier results.
- Figure 13 arranges each validation query beside its three nearest training-set neighbors, ordered from nearest to third nearest.
Appendix D Learning curves
Learning curves track generation and representation metrics across latent-space, loss, capacity, resolution, architecture, and optimization ablations, revealing substantial differences among variants.
- Latent distribution Pz and stochastic E: Uniform Pz reaches 31.63, while Deterministic E reaches 31.21 and Base reaches 31.40 in the reported learning-curve results.
- Unary loss terms: No Unaries reaches 43.64 and 42.70 on reported metrics, while z Unary Only reaches 40.32 and 39.02 and x Unary Only reaches 32.07 and 31.79.
- G capacity: Small G (32) reaches 60.85 and 60.67, whereas Small G (64) reaches 38.81 and 38.56 and Base reaches 31.40 and 30.85.
- High resolution E with varying resolution G: High Res E with High Res G reaches 38.21, compared with 27.91 and 27.74 for High Res E and 15.77, 15.68, and 15.64 for High Res E with Low Res G.
- E architecture: E architecture variants report values from 26.68 for High Res E, RevNet ×2 to 26.18 for High Res E, RevNet, with RevNet ×4 at 26.59.
- Decoupled E/G learning rates: With decoupled optimization and a 10× higher E learning rate, High Res E reaches 27.17 and RevNet ×4 reaches 27.02, versus 26.25 and 26.59 without the increase.