PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Models as a Data Source for Multiview Representation Learning

Ali Jahanian, Xavier Puig, Yonglong Tian, Phillip Isola

arXiv:2106.05258v3cs.CV

TL;DR

This paper asks whether black-box generative models can replace real datasets for visual representation learning. It learns representations from generated multiview data and finds that they can rival or sometimes outperform representations learned from real datasets.

  • Problem

    The paper investigates whether representation learning can use synthetic samples from black-box generative models instead of directly accessing real datasets.

  • Method

    The method trains representation learners on generated images, using nearby latent-space points to create multiple views of related content and contrastive training pairs.

  • Results

    Representations learned from large quantities of generative data rival and sometimes outperform those learned from real datasets, while latent-space views improve performance beyond pixel-space transformations alone.

  • Takeaways & Limitations

    Generative models can serve as compressed, organized sources for visual representation learning, although effective use depends on sampling and training choices.

  • Takeaways & Limitations

    The approach is studied with implicit generative models and faces privacy and bias risks that require auditing and mitigation.

Abstract

from arXiv · show

Generative models are now capable of producing highly realistic images that look nearly indistinguishable from the data on which they are trained. This raises the question: if we have good enough generative models, do we still need datasets? We investigate this question in the setting of learning general-purpose visual representations from a black-box generative model rather than directly from data. Given an off-the-shelf image generator without any access to its training data, we train representations from the samples output by this generator. We compare several representation learning methods that can be applied to this setting, using the latent space of the generator to generate multiple "views" of the same semantic content. We show that for contrastive methods, this multiview data can naturally be used to identify positive pairs (nearby in latent space) and negative pairs (far apart in latent space). We find that the resulting representations rival or even outperform those learned directly from real data, but that good performance requires care in the sampling strategy applied and the training method. Generative models can be viewed as a compressed and organized copy of a dataset, and we envision a future where more and more "model zoos" proliferate while datasets become increasingly unwieldy, missing, or private. This paper suggests several techniques for dealing with visual representation learning in such a future. Code is available on our project page https://ali-design.github.io/GenRep/.

1 INTRODUCTION

This paper asks whether effective visual representations can be learned using only samples from a trained generative model, without its original dataset. It studies latent-space multiview learning and finds that generative samples can match or exceed real-data representations under suitable choices.

  • Generative models offer compressed, shareable synthetic data that can reduce dataset distribution and privacy constraints, although models may still inherit or amplify data risks.
  • The paper studies whether effective visual representations can be learned from a trained generative model without access to its training dataset.
  • The framework compares representation learning from real and generated images, with and without labels, and evaluates transfer performance on held-out datasets and tasks.
  • Contrastive learning creates multiple views by transforming generated samples in latent space, using nearby latent points to represent similar semantic content.
  • Combining latent-space and pixel-space transformations improves performance over either transformation alone.
  • With sufficiently high-quality generators, learned representations can outperform those learned directly from real data, while performance improves sub-logarithmically with more generated samples.

2 RELATED WORK

Prior work uses synthetic and generative data for augmentation, classification, segmentation, and representation learning, but this paper focuses on general visual representations learned solely from a black-box generator. Its setting differs especially by avoiding access to the original real data and generator internals.

  • Synthetic data has been used for privacy preservation, alternative sampling, semantic segmentation, and reducing the domain gap between generated and real images.
  • GAN-generated images can support classifier training, but this paper targets more general visual representation learning, mainly with unlabeled samples from unconditional generators.
  • Some related contrastive-adversarial methods require real data during training, whereas this work studies representation learning without access to real data.
  • Other StyleGAN-based approaches use generator internals as image representations, while this paper treats the generator as a black-box source of downstream training data.
  • Contrastive learning relies on positive and negative pairs, and prior work emphasizes that well-chosen image transformations are central to its success.
  • Generative representation learning traditionally models the data distribution, whereas this paper uses a generative model to produce training data for a separate representation encoder.

3 METHOD

The method learns visual representations from a generative model instead of a dataset by using latent- and pixel-space transformations to construct views for contrastive learning. It also compares these approaches with non-contrastive generator inversion.

  • 3 METHOD: The proposed setting takes an implicit generative model G instead of a dataset and learns an encoder F from generated images, including class-conditional variants.The framework applies to latent-variable generative models; experiments investigate GANs.
  • 3.1 CONTRASTIVE LEARNING FRAMEWORK: Contrastive learning uses positive and negative image pairs, pulling views of the same scene together while pushing views of different scenes apart.The framework focuses on how sampling and transformations define these pairs.
  • 3.2 CONTRASTIVE LATENT VIEWS + PIXEL TRANSFORMATIONS: For generated data, latent transformations create positive views by applying Tz to sampled latents, optionally followed by pixel transformations Tx for both anchor and positive images.Negatives are generated from independently sampled latent vectors under the same transformation scheme.
  • 3.3 CREATING VIEWS WITH Tx AND Tz: The method treats transformations as data views and uses standard SimCLR pixel augmentations alongside newly explored latent-space transformations for contrastive representation learning.The paper identifies latent-space transformations as a new setting for contrastive learning.
  • 3.3.1 GAUSSIAN LATENT VIEWS: Nearby latent points are used as semantically related views through truncated Gaussian offsets, Tz(z) = z + wGauss.The offset distribution is a truncated Normal distribution, and the framework also studies learned semantic steering vectors.
  • 3 METHOD: As a non-contrastive alternative, the inverter learns an encoder intended to use the generator’s latent representation as an embedding of the generated image.This approach is presented alongside contrastive learning methods.

4 EXPERIMENTS

Experiments evaluate representation learning from unconditional generative models using latent and pixel transformations, contrastive and non-contrastive methods, and varying sample and view-generation strategies. Latent views improve contrastive learning, Gaussian transformations perform strongly, and StyleGAN2-generated data can outperform direct training on real photographs.

  • 4.1.1 EFFECT OF THE LATENT AND IMAGE TRANSFORMATIONS: Pixel transformations produce strong representations, and combining them with latent transformations substantially improves transfer performance across unconditional generative models.The comparison includes Gaussian and steering transformations applied alongside pixel-space augmentations.
  • 4.1.1 EFFECT OF THE LATENT AND IMAGE TRANSFORMATIONS: Gaussian latent transformations perform similarly or better than designed steering transformations despite perturbing all latent directions randomly.This finding holds across the compared unconditional generative-model experiments.
  • 4.1.1 EFFECT OF THE LATENT AND IMAGE TRANSFORMATIONS: StyleGAN2-generated data outperforms training directly on LSUN CAR photographs, while adding latent transformations further boosts performance beyond pixel transformations alone.The authors attribute the result partly to StyleGAN2’s realism and diversity, reporting FID = 2.32.
  • 4.1.2 EFFECT OF THE DISTANCE BETWEEN LATENT VIEWS: Linear transfer performance follows an inverse-U-shaped curve as the Gaussian latent-view distance changes, indicating an intermediate sampling scale is preferable.The experiment varies the Gaussian transformation standard deviation and evaluates transfer on ImageNet100.
  • 4.2 NON-CONTRASTIVE METHODS: The study compares contrastive learning with an inverter that predicts latent codes, and the non-contrastive approach performs poorly relative to contrastive methods.The inverter is trained on 1.3M image–latent-code pairs generated by the model.
  • 4.3 EFFECT OF THE NUMBER OF SAMPLES: Sample-count experiments vary the number of unique generated examples while keeping encoder training iterations fixed, measuring linear classification on ImageNet100.The evaluation uses Gaussian views combined with pixel transformations.

5 CONCLUSION

The paper concludes that black-box generative models can supply data for visual representation learning through nearby latent-space views. Representations from large quantities of generated data rival and sometimes outperform those learned from real datasets.

  • 5 CONCLUSION: Figure 6 evaluates how the number of generated samples affects linear transfer to ImageNet100 for Gaussian views.“Gaussian” denotes views produced by Tz = z + wGauss.
  • 5 CONCLUSION: Latent-space views from black-box implicit generative models improve representation learning beyond pixel-space transformations alone.Nearby latent points generate multiple views of similar image content for contrastive and other multiview methods.
  • 5 CONCLUSION: Representations learned from large quantities of generative data rival and sometimes outperform representations learned from real datasets.The conclusion frames generative models as a potential source of training data for vision systems.

6 ETHICS STATEMENT

The ethics discussion identifies privacy, bias, and data-governance benefits associated with generated data, while emphasizing that generative models introduce their own risks. Responsible use therefore requires auditing models and studying mitigation strategies.

  • 6 ETHICS STATEMENT: Generated data may help protect privacy and usage rights, remove sensitive attributes, and reduce biases present in real datasets.The paper presents these as potential societal advantages of learning representations from implicit generative models.
  • 6 ETHICS STATEMENT: Generative models can reveal training data or amplify dataset biases, creating privacy and societal risks if they are not properly audited.The authors connect these risks to potential negative impacts in inappropriate contexts.
  • 6 ETHICS STATEMENT: Using generated images requires attention to preprocessing choices, including center cropping, removal of LSUN CAR padding, and reconstruction of unavailable real-data comparisons.These implementation procedures define the experimental image inputs and comparison data.
  • 6 ETHICS STATEMENT: Steering views encode shifts, zoom, rotations, and color changes through latent walks whose magnitudes are controlled by transformation-specific coefficients.The steering construction combines learned walks for several pixel-space transformations.
  • 6 ETHICS STATEMENT: In class-conditional settings, labels define positive and negative examples, with one same-latent positive receiving a different pixel transformation.The method also samples other positives independently from the latent distribution while preserving class conditioning.
  • 6 ETHICS STATEMENT: Independent latent views are formed by drawing a new random latent code, sharing no image information in the unconditional setting but preserving class semantics when conditioned.The paper describes this as a naive unconditional baseline and a sensible class-conditional strategy.

C.2 EXPERIMENTS

The experiments evaluate contrastive representation learning from class-conditional BigGAN samples using latent and pixel-space views. Latent view generation is combined with supervised contrastive learning and evaluated through transfer classification.

  • C.2 EXPERIMENTS: Latent views are created with Gaussian or steer transformations, independent class-conditioned samples, and optional pixel transformations.The experiments generate anchor images, neighbor views, and combined latent-plus-pixel views from BigGAN.
  • C.2.2 EFFECT OF THE LATENT AND IMAGE TRANSFORMATIONS: Adding latent transformations substantially improves pixel-augmented contrastive representations for unconditional IGMs but yields only marginal gains for class-conditional IGMs.The authors attribute the smaller conditional improvement to class-label supervision already providing strong semantic representations.

C.2.3 NON-CONTRASTIVE METHODS

The paper compares non-contrastive inversion and classification objectives with contrastive learning for representations from class-conditional generative samples. Inversion performs poorly, while classification approaches supervised contrastive performance.

  • C.2.3 NON-CONTRASTIVE METHODS: The inverter performs poorly compared with contrastive methods when learning representations from class-conditional IGMs.It predicts latent codes and class labels using reconstruction and auxiliary classification losses.
  • C.2.3 NON-CONTRASTIVE METHODS: 65.2% ImageNet100 Top-1 accuracy is achieved by the classifier, versus 66.8% for supervised contrastive learning with pixel transformations.The classifier uses a ResNet-50 backbone, pixel augmentations, and a softmax cross-entropy objective.
  • C.2.3 NON-CONTRASTIVE METHODS: The classifier is a restricted form of the inverter that predicts class labels but avoids latent-code reconstruction.Pixel augmentations can be applied in the classifier setup and are reported to support higher performance.

C.2.4 EFFECT OF THE NUMBER OF SAMPLES

The sample-count experiment asks how much synthetic data is needed for effective coverage and visual representations. Performance improves as the number of unique samples increases, but only sub-logarithmically.

  • C.2.4 EFFECT OF THE NUMBER OF SAMPLES: The experiment evaluates linear classification on ImageNet100 while increasing the number of unique samples and holding training iterations constant.With more images, the encoder revisits each seen image fewer times.
  • C.2.4 EFFECT OF THE NUMBER OF SAMPLES: Performance increases with more synthetic samples, but the improvement is sub-logarithmic.The experiment varies unique samples for Gaussian views combined with pixel transformations while keeping encoder training iterations fixed.
  • C.2.4 EFFECT OF THE NUMBER OF SAMPLES: The findings are consistent with a small generalization gap between infinite-data online learning and a sufficiently large offline regime.This comparison is presented as consistency with recent work rather than as a direct measurement of the same gap.

D FURTHER TRAINING DETAILS AND COMPARISON TO SIMCLRV1

The section explains why the reported real-data SimCLR baseline trails prior results and examines the effects of resolution, optimizer, and training time. These choices materially affect transfer accuracy.

  • D FURTHER TRAINING DETAILS AND COMPARISON TO SIMCLRV1: 57.5%/62.8% are the comparable prior SimCLR results, versus 43.9% reported here under the same batch-size comparison.The section identifies differences in image resolution, linear-evaluation optimizer, and training duration as explanations for the gap.
  • D FURTHER TRAINING DETAILS AND COMPARISON TO SIMCLRV1: 50.3% ImageNet1000 accuracy is obtained by evaluating the same encoder with 256 × 256 images instead of 128 × 128 images.The corresponding BigBiGAN result with SimCLR pixel and Gaussian latent augmentations is 49.7%.
  • D FURTHER TRAINING DETAILS AND COMPARISON TO SIMCLRV1: Changing the linear-evaluation optimizer from LARS to SGD is associated with a drop from 62.8% to 57.5%.The cited comparison comes from the prior work’s experiments and is reported as an optimizer effect.
  • D FURTHER TRAINING DETAILS AND COMPARISON TO SIMCLRV1: SimCLR transfer accuracy rises from 15.98% at 5 epochs to 32.48% at 15 epochs under the reported setup.The intermediate 10-epoch result is 23.66%.

E FURTHER TRANSFER LEARNING TASKS

The paper evaluates representations beyond classification, including transfer learning on ImageNet100 and object detection on PASCAL VOC. Results are reported separately for unconditional and class-conditional image generative models.

  • E FURTHER TRANSFER LEARNING TASKS: The lighter-weight protocol trains and evaluates representations using data at the ImageNet100 scale.This setting is described as the protocol used in the main text's Section 4.
  • E FURTHER TRANSFER LEARNING TASKS: Object detection uses Faster-RCNN with an R50-C4 architecture and fine-tuning on PASCAL VOC trainval07+12.The evaluation reports AP, AP50, and AP75 under standard COCO metrics.
  • E FURTHER TRANSFER LEARNING TASKS: Results for unconditional and conditional image generative models are reported in Tables 4 and 5.The supplied text identifies the tables but does not provide their numerical entries.

F MIXING REAL AND SYNTHETIC IGM DATA

The paper examines replacing portions of real training data with synthetic image-generative-model samples and visualizes latent-view construction. A small synthetic fraction can help, whereas performance decreases as real data is reduced further.

  • F MIXING REAL AND SYNTHETIC IGM DATA: Tables 4–6 distinguish real-data results from synthetic-data results across unconditional and class-conditional image-generative-model settings.The captions specify the generators, transformations, and objectives, including contrastive, supervised contrastive, and classifier objectives.
  • F MIXING REAL AND SYNTHETIC IGM DATA: Figure 9 plots the percentage of real data replaced by synthetic data against Top-1 accuracy.The figure concerns augmentation of real data with synthetic IGM data.
  • F MIXING REAL AND SYNTHETIC IGM DATA: 5% synthetic images is a sweet spot when replacing ImageNet100 images with BigBiGAN samples.The experiment trains SimCLR ResNet-50 models and evaluates learned features with a linear classifier using Top-1 accuracy.
  • F MIXING REAL AND SYNTHETIC IGM DATA: Performance decreases as the number of real images decreases in the real–synthetic mixing experiment.The supplied passage reports this trend while also identifying a small-synthetic-data sweet spot.
  • F MIXING REAL AND SYNTHETIC IGM DATA: Figures 10 and 11 provide qualitative latent-view examples for unconditional and class-conditional image generative models.The examples compare anchors with Gaussian and steering neighbors for BigBiGAN, StyleGAN LSUN CAR, and BigGAN settings.
Loading 2106.05258v3…