Large Scale Adversarial Representation Learning

• Prior BiGAN representation learning [7] was effective on ImageNet, but its DCGAN generator [31] could not produce high-quality images; BigGAN [2] motivates revisiting the approach.The authors describe BigGAN as capable of capturing many ImageNet modes and much of its structure.Source: §1 · p. 2
• BigBiGAN (BiGAN with a BigGAN generator) matches the state of the art in unsupervised representation learning on ImageNet.Source: §1 · p. 2
• The authors propose a more stable version of BigBiGAN’s joint discriminator.Source: §1 · p. 2
• The work provides a thorough empirical analysis and ablation study of model design choices.Source: §1 · p. 2
• The representation-learning objective also improves unconditional image generation, achieving state-of-the-art results in unconditional ImageNet generation.Source: §1 · p. 2
• The authors open-source pretrained BigBiGAN models on TensorFlow Hub.Source: §1 · p. 2

2 BigBiGAN

• The encoder E predicts latents from data, while generator G maps sampled latents to data, modeling inverse conditional distributions.The encoder models P(z|x), and the generator models P(x|z).Source: §2 · p. 3
• A joint discriminator distinguishes encoder-produced (x, z) pairs from generator-produced pairs, while E and G try to make the two joint distributions indistinguishable [7] [10].With deterministic E and G, the global optimum makes the functions inverses, such as x = G(E(x)) for data x in the support of Px [7].Source: §2 · p. 3
• BigBiGAN adopts BigGAN’s generator and discriminator architectures [2] and adds unary data- and latent-only loss terms to improve representation learning without compromising generation.The data-only term matches the original GAN objective and steers the generator to match the image distribution independently of latent inputs.Source: §2 · p. 3
• The discriminator combines unary data and latent scores with a joint score; its submodules F and H process images and flat latent vectors, respectively, while J computes the joint score.The losses use a hinge function to regularize the discriminator, as in BigGAN [2].Source: §2 · p. 3
• The alternative discriminator loss that applies the hinge to the sum of all three terms performed significantly worse than separately clamping each term.The separately clamped loss is the one defined for BigBiGAN.Source: §2 · p. 3

3 Evaluation

• Evaluation freezes a BigBiGAN trained on unlabeled ImageNet and fits a fully supervised linear classifier using all training labels; generation is measured with IS and FID.The classifier protocol follows [41]; IS and FID are the standard generation metrics cited as [35] and [18].Source: §3 · p. 4

3.1 Ablation

• The non-deterministic encoder improves classification over deterministic E and uniform Pz, without a generation cost; the x unary term strongly improves generation, while z unary effects are smaller and mixed.The z term slightly improves classification and FID, especially without the x term, but worsens IS.Source: §3.1 · pp. 4–5
• With E and D fixed, reducing generator capacity to one-third (Small G (32)) significantly worsened classification; at two-thirds (Small G (64)), generation was substantially worse and classification modestly worse.The reported model performed reasonably for generation earlier in training, reaching IS 14.69 and FID 60.67.Source: §3.1 · p. 5
• Raising E input resolution to 256 × 256 improves representation results as G resolution rises to 256 × 256, and improves generation—especially FID—even when G remains at 128 × 128.Because training with G at 256 × 256 is much slower, subsequent results use 128 × 128 G.Source: §3.1 · p. 5
• A 10× higher learning rate for E accelerates training and improves final representation results; the ResNet-50 linear-classifier accuracy rises by nearly 3%, with similar gains for RevNet-50 ×4.The wider RevNet ×4 architecture also improves on smaller encoder variants and is used for final results.Source: §3.1 · pp. 5–6

3.2 Comparison with prior methods

• 60.8% top-1 accuracy improves on the 55.4% result from rotation-prediction pretraining [24], establishing BigBiGAN as state of the art among recent unsupervised results.The comparison uses the same representation-learning architecture and feature; AvePool matches concurrent CPC results [17].Source: §3.2 · pp. 6–7, Table 2 · p. 7
• BigBiGAN significantly improves IS and FID over unconditional BigGAN with the same fixed label; high-resolution E surpasses BigGAN + Clustering on both metrics, with further gains at 1M steps.Generation remains at 128 × 128 with high-resolution E; these results use lighter data augmentation than the Table 1 ablations.Source: Table 3 · p. 7, §3.2 · p. 8

3.3 Reconstruction

• Reconstructions are not pixel-perfect because the objective enforces no explicit reconstruction cost, but often retain category-level semantics, pose, position, and texture rather than low-level details.Reconstructions are computed as G(E(x)) and are not computed during training.Source: §3.3 · p. 8, Figure 2 · p. 8

4 Related work

• Self-supervised image learning includes relative-location prediction [5], CPC [39] [17], colorization [41] [42], motion segmentation [30], rotation prediction [11] [4], GAN discrimination [31] [4], and exemplar matching [8].Rigorous empirical comparisons of several such approaches have also been conducted [6] [24].Source: §4 · pp. 8–9
• Unlike most self-supervised approaches, generative-model methods can use full-resolution inputs without cropping or modification, allowing representations to apply directly downstream without domain shift.Cropping or other data modification may still be beneficial as augmentation.Source: §4 · p. 9
• ACNs [15] condition dataset-level compression on similar prior codes, while VQ-VAEs [40] pair vector-quantized encoders with autoregressive decoders.ACNs can “daydream” semantically similar samples; VQ-VAEs produce faithful reconstructions with high compression and show representation-learning results in reinforcement-learning settings.Source: §4 · p. 9
• Adversarial autoencoders replace VAE prior KL regularization with a discriminator, while VAE-GAN [25] uses discriminator feature-space rather than pixel-space reconstruction.AGE [38] and α-GAN [33] add encoders to stabilize GAN training.Source: §4 · p. 9
• Unlike these variants, BiGAN [10] [7] trains neither encoder nor generator with an explicit reconstruction cost.Although (Big)BiGAN implicitly minimizes reconstruction cost, qualitative reconstructions suggest it emphasizes high-level semantics over pixel-level detail.Source: §4 · p. 9

5 Discussion

• BigBiGAN, an approach based purely on generative models, achieves state-of-the-art unsupervised image representation learning on ImageNet.Source: §5 · p. 9
• The ablation study supports the potential for powerful generative models to benefit representation learning and for inference models to improve large-scale generative models.Source: §5 · p. 9
• Future progress may come from advances in generative and inference models, as well as scaling to larger image databases.Source: §5 · p. 9

Appendix A Model and optimization details

• BigBiGAN uses Adam with batch size 2048, two discriminator updates per joint generator–encoder update, spectral normalization in G and D, and EMA with decay 0.9999 for G and E.Orthogonal regularization was omitted because it worsened unconditional results; EMA gives only a small improvement for E evaluations but a substantial one for G evaluations. Full cross-replica batch normalization is used in G and E.Source: Appendix A · p. 12
• For training and linear-classification evaluation, inputs use ResNet-style augmentation with crop size 128 or 256 rather than 224.Table 2 extends BigBiGAN training to 1M steps and evaluates on the official 50K-image validation set; classifier training runs for 100K Adam steps across five learning rates, with selection and early stopping based on trainval.Source: Appendix A · p. 12
• For Table 1 ablations, models run for 500K steps with early stopping on trainval; linear classifiers start at zero and train for 5K Adam steps at learning rate 0.01 with EMA decay 0.9999.Trainval is a 10K-image holdout from the official ImageNet training set, and classifiers are periodically retrained from scratch to monitor representation learning.Source: Appendix A · p. 12

Supervised model performance

• Table 4 reports ImageNet validation accuracy for fully supervised end-to-end training of the architectures used in the representation-learning experiments.FID is reported against full ImageNet training-set statistics, with validation-set FID also reported in Table 3 for comparison with [27].Source: Appendix A · p. 12, Table 4 · p. 12

First layer convolutional filters

• With a higher encoder learning rate, RevNet ×4 encoder filters appear less noisy and more Gabor-like and colorful, suggesting input-layer filters can diagnose undertrained models.Figure 3 visualizes the 1024 first-layer filters of the RevNet ×4 encoder; the comparison corresponds to the RevNet ×4 and RevNet ×4 (↑E LR) rows in Table 1.Source: Appendix A · p. 12, Figure 3 · p. 13

Samples and reconstructions

• Samples were selected by best FID versus training-set statistics, with IS and FID reported, while reconstructions were selected by lowest relative pixel-wise ℓ1 error.The appendix presents examples across Base, lighter-augmentation, and high-resolution configurations.Source: Appendix B · p. 13
• The appendix shows 128 × 128 samples and real-input/reconstruction pairs for Base and lighter-augmentation BigBiGAN configurations.The lighter-augmentation configuration uses [27].Source: Appendix B · p. 13, Figure 4 · p. 15, Figure 5 · p. 16
• The relative ℓ1 metric compares reconstruction error with a random-input baseline, is approximately 1 for randomly initialized G and E, and penalizes degenerate outputs such as the mean image.Its denominator is estimated using N = 50K data samples, comparing each sample with one neighboring sample.Source: Appendix B · p. 13

Iterated reconstruction

• After dozens or hundreds of reconstruction iterations, little of the original input apparently remains, although the first several steps often retain some semantics.Figure 12 shows iterations from R1 through R9 and at intervals up to R500.Source: Appendix B · pp. 14, 23, Figure 12 · p. 23

Samples and reconstructions

• High-resolution examples include 128 × 128 samples and reconstructions from 256 × 256 encoder inputs, plus 256 × 256 samples and reconstructions.The 256 × 256 encoder-input reconstructions are upsampled from 128 × 128 for visualization.Source: Figure 8 · p. 19, Figure 9 · p. 20, Figure 10 · p. 21

Appendix C Nearest neighbors

• Evaluation uses the best-performing RevNet ×4, ↑E LR model’s global-average-pooling 8192D features, with center crops resized to 256 × 256 and no data augmentation.Both training and validation images use this preprocessing.Source: Appendix C · p. 24
• For k > 1, the predicted label is the plurality vote among neighbors; ties are broken using k=1 classification among the tied labels.The metric is a normalized ℓ1 or ℓ2 distance; D2 corresponds to cosine distance.Source: Appendix C · p. 24

Quantitative results

• With D1, top-1 accuracy is around 38% at k=1 and 43% at k=25, then drops slightly at k=50; D1 outperforms D2 across all tested k.The tested values are k ∈ {1, 5, 25, 50}.Source: Appendix C · p. 24

Qualitative results

• In unsupervised retrievals, neighbors often match query objects’ high-level semantics; pose, position, background, or color scheme also appear to influence matches.For example, the nearest neighbors of an RV query are all RVs facing roughly the same direction.Source: Appendix C · p. 24, Figure 13 · p. 25

Discussion

• Nearest-neighbor classification is far below ImageNet state-of-the-art and the paper’s linear-classifier results, but predicts labels without supervised learning of model parameters.The authors propose it as an additional benchmark for future unsupervised representation-learning approaches.Source: Appendix C · p. 24

Appendix D Learning curves

• The curves cover image generation (Figure 14) and comparisons of latent-space choices and unary-loss components (Figures 15–16).The latent-space variants are Base, Deterministic E, and Uniform Pz; the unary-loss variants are Base, x Unary Only, z Unary Only, and No Unaries (BiGAN).Source: Appendix D · p. 26, Figure 15 · p. 27, Figure 16 · p. 28
• Additional curves compare generator capacities and high-resolution encoders paired with generators at varying resolutions (Figures 17–18).The listed variants include Small G (32), Small G (64), High Res E (256), Low Res G (64), and High Res G (256).Source: Appendix D · p. 26, Figure 17 · p. 29, Figure 18 · p. 30
• The appendix also plots encoder-architecture comparisons and the effect of training E with a 10× higher learning rate than G (Figures 19–20).The architecture comparisons include ResNet and RevNet variants; the learning-rate plots include ResNet (↑E LR) and RevNet ×4 (↑E LR).Source: Appendix D · p. 26, Figure 19 · p. 31, Figure 20 · p. 32