Source-linked AI summary
StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learners
Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, Dilip Krishnan
TL;DR
The paper studies whether synthetic images from text-to-image models can support large-scale visual representation learning instead of real-image collection. It uses Stable Diffusion and introduces StableRep, which contrasts multiple images generated from the same caption, finding synthetic representations competitive with or superior to real-data baselines.
Problem
The paper asks whether text-to-image models can provide practical synthetic data for large-scale visual representation learning, given the difficulty and bias risks of collecting real images.
Method
StableRep generates multiple Stable Diffusion images per caption and treats them as positives in a multi-positive contrastive learning objective.
Results
StableRep surpasses SimCLR and CLIP trained with corresponding real images on large-scale representation benchmarks, including 76.7% ImageNet linear accuracy using solely synthetic images.
Takeaways & Limitations
Synthetic images can train powerful visual representations while reducing reliance on collecting large quantities of real images.
Takeaways & Limitations
The effectiveness of synthetic-image self-supervision is not fully understood, generation is slow, and prompt-image semantic mismatch remains unaddressed.
Abstract
from arXiv · showhide
We investigate the potential of learning visual representations using synthetic images generated by text-to-image models. This is a natural question in the light of the excellent performance of such models in generating high-quality images. We consider specifically the Stable Diffusion, one of the leading open source text-to-image models. We show that (1) when the generative model is configured with proper classifier-free guidance scale, training self-supervised methods on synthetic images can match or beat the real image counterpart; (2) by treating the multiple images generated from the same text prompt as positives for each other, we develop a multi-positive contrastive learning method, which we call StableRep. With solely synthetic images, the representations learned by StableRep surpass the performance of representations learned by SimCLR and CLIP using the same set of text prompts and corresponding real images, on large scale datasets. When we further add language supervision, StableRep trained with 20M synthetic images achieves better accuracy than CLIP trained with 50M real images.
1 Introduction
The paper asks whether text-to-image models can replace costly, noisy real-image collection for large-scale visual representation learning. Using Stable Diffusion, it finds synthetic-data pre-training effective and introduces StableRep, which leverages multiple generations from each caption.
- Real-image collection is costly, noisy, and can introduce domain gaps and societal biases.
- Text-to-image models could generate requested images from natural-language commands at very low cost.
- Stable Diffusion generates synthetic images from captions in large-scale image-text datasets such as CC12M and RedCaps.
- With suitable classifier-free guidance, self-supervised learning on synthetic images can match or exceed training on same-size real-image datasets.
- StableRep treats multiple images generated from one caption as positives, achieving 76.7% ImageNet linear accuracy with solely synthetic images.
- StableRep trained with 20M synthetic images achieves better accuracy than CLIP trained with 50M real images when language supervision is included.
2 Standard Self-supervised Learning on Synthetic Images
The study evaluates standard self-supervised methods on Stable Diffusion images and examines how guidance scale affects representation quality. Synthetic images can outperform or match real images, although the best guidance scale depends on the learning method.
- A text-to-image generator maps text and latent noise to images, allowing representation learning from generated rather than real-image datasets.
- Stable Diffusion uses classifier-free guidance, whose scale trades off synthesized-image diversity and quality.
- For SimCLR, guidance scale 8 or 10 is optimal, with each achieving 62.0% ImageNet linear-probing accuracy.
- Synthetic SimCLR images at guidance scale 8 reach 62.0% accuracy versus 60.4% for paired real images.
- For MAE, guidance scale 6 is optimal, and synthetic images yield 4.2% better accuracy than real images.
- After ImageNet fine-tuning, synthetic MAE images at guidance scale 6 reach 82.9% versus 82.6% for real images.
- StableRep’s pipeline generates multiple caption-conditioned images, augments them, and treats them as positives in a multi-positive contrastive loss.
- Across additional methods, synthetic images improve MAE, DINO, and SimCLR, match BYOL, and perform slightly worse for MoCo-v3.
3 Multi-Positive Contrastive Learning with Synthetic Images
StableRep uses multiple Stable Diffusion images generated from each caption as mutual positives, training representations with a multi-positive contrastive loss rather than relying on image class labels.
- Text-to-image models generate diverse images from one caption, providing multiple semantically similar positive samples that are difficult to collect at scale.Different reverse-diffusion noise vectors produce the multiple samples.
- The method treats an encoded anchor and candidate embeddings as a matching problem, with q representing the anchor’s probability of matching each candidate.The candidates are evaluated using a K-way softmax distribution.
- The ground-truth distribution p assigns probability across all candidates matched to the anchor, while 1match indicates whether each pair matches.Matching is based on images generated from the same caption.
- The multi-positive contrastive loss is the cross-entropy between the matching distribution p and contrastive distribution q.This generalizes single-positive contrastive learning, where p is one-hot, without requiring image class labels.
- Algorithm 1 processes batches containing m images for each of n captions, applies augmentation, and computes encoder similarities with self-masking.The implementation forms logits from normalized representations and averages the cross-entropy loss across the batch.
4 Experiments
Experiments show that StableRep learns strong representations from synthetic images across classification, few-shot recognition, and segmentation, while ablations identify effective generation and training choices.
- Main results: StableRep achieves the highest accuracy on all 11 image classification datasets spanning different domains.
- Main results: StableRep stands out on 9 of 10 datasets in 5-way, 5-shot image classification.The evaluation uses frozen representations with a simple classifier.
- Main results: StableRep pretrained on synthetic data outperforms MAE pretrained on real ImageNet images for ADE20k semantic segmentation.Models are fine-tuned with UperNet after 35 or 105 SimCLR-equivalent pretraining epochs.
- Ablation analysis: Generating more than one image per caption improves performance under a fixed image budget, with l = 8 improving over l = 1 by 4.8%; gains saturate near l = 10.The final experiments generate 10 images per caption.
- Ablation analysis: StableRep is robust to the number of images sampled per caption, with a peak of 69.8% linear probing accuracy at m = 8; m = 6 is used by default.The comparison varies m from 2 to 10 while keeping batch size fixed.
- Ablation analysis: Small classifier-free guidance scales give the best StableRep transfer accuracy, likely because they produce greater within-caption variation.This differs from SimCLR, whose accuracy peaks at larger guidance scales.
- Ablation analysis: Scaling the backbone to ViT-L/16 improves ImageNet linear probing accuracy by 1.9% and average fine-grained classification accuracy by 0.7%.ViT-L/16 pretraining was unstable because the loss exploded to NaN.
5 Adding Language Supervision
Language supervision improves StableRep’s ImageNet performance, while synthetic-data models show fairness gains but unresolved compositionality and text-image alignment issues.
- Language-supervised CLIP: CLIP with synthetic images reaches 34.9% zero-shot ImageNet accuracy at guidance scale w = 2, below the 40.2% real-image baseline.The gap may reflect semantic misalignment between prompts and generated images, especially for fine-grained classes.
- Language supervision: StableRep+ improves ImageNet linear probing from 72.8% to 74.4% on CC12M and from 73.7% to 75.4% on RedCaps.StableRep+ adds image-to-text and text-to-image contrastive losses to StableRep.
- Language supervision: StableRep+ with 10M captions outperforms CLIP with 50M captions, yielding 5x caption efficiency and 2.5x image efficiency.The comparison uses synthetic images for StableRep+ and real images for CLIP.
- Fairness: Synthetic training improves worst-class accuracy on FairFace, but a geographic bias remains across all models.For example, StableRep+ raises one CC12M class from 0.3% to 27.2% and one RedCaps class from 0.4% to 22.8%.
- Compositionality: Compositionality results are mixed: synthetic data slightly improves relational understanding on CC12M but decreases accuracy on RedCaps.The paper calls for further in-depth investigation.
6 Related Work
The paper builds on text-to-image diffusion, visual representation learning, and prior uses of synthetic data across computer-vision tasks.
- Text-to-image generative models: Text-to-image models generate rich and diverse images from large image-text pairs, with Stable Diffusion representing the diffusion-based models leveraged here.Stable Diffusion operates as a latent-space denoising diffusion model.
- Visual representation learning: Visual representation learning includes pretext tasks, masked image modeling, and contrastive learning, among other approaches.The paper selects SimCLR and MAE because of their simplicity and strong performance.
- Learning from synthetic data: Synthetic images have supported computer-vision training for optical flow, autonomous driving, segmentation, detection, pose estimation, and classification.The paper positions its work within this broader synthetic-data literature while focusing on representation learning.
7 Conclusion, Limitations and Broader Impact
The paper concludes that synthetic images can train strong visual representations, while identifying unresolved questions about effectiveness, generation cost, semantic mismatch, bias, attribution, and domain transfer.
- Conclusion: StableRep combines Stable Diffusion’s stochasticity with a multi-positive contrastive loss to surpass representations trained on real data alone.Synthetic pretraining performs strongly across linear probing and few-shot classification, while vanilla self-supervised methods can also match or outperform real-data training.
- Limitations: The paper does not explain why synthetic images can outperform equal-sized real datasets and notes that the finding may depend on its evaluation methodology.This limits how broadly the observed advantage should be interpreted.
- Limitations: Synthetic-data generation remains slow, preventing StableRep training with non-repetitive images synthesized online.The reported generation time is approximately 0.8 seconds per image on A100 and 2.2 seconds on V100 GPUs.
- Broader impacts: Prompt choice, uncurated web data, and generative-model behavior can preserve or exacerbate social biases in synthetic datasets.Image attribution is also challenging for synthetic data.
- Domain transfer: Random Downsample addresses the domain gap caused by constant high-resolution synthetic images when transferring to low-resolution datasets.It randomly downsamples to 64 or 128 before resizing back to 224.
- Domain transfer: Random Downsample significantly improves CIFAR-10 and CIFAR-100 performance while maintaining performance on other datasets.The augmentation is more beneficial on average for synthetic-image pretraining than real-image pretraining.
A.3 StableRep pre-training
StableRep uses its own pretraining settings, largely aligned with SimCLR but adjusted for its single-crop batch construction and computational accounting.
- StableRep pre-training: StableRep uses the same hyperparameters as SimCLR, except its base learning rate is defined for 512 images rather than 256.Each StableRep image has one single crop, and training uses a batch size of 8256 images across 32 GPUs.
- StableRep pre-training: StableRep computation is reported in SimCLR-equivalent epochs.This provides the accounting convention used for comparing its training schedule with SimCLR.
A.4 CLIP training
This section specifies CLIP training and evaluation procedures, including encoder settings, ImageNet linear probing, and fine-grained classification protocols.
- CLIP training: CLIP trained on CC12M achieves 40.2% ImageNet zero-shot accuracy, compared with 36.0% reported using the same architecture.
- Evaluation: ImageNet linear probing uses a frozen representation with a regularized multinomial logistic regression classifier and no weight decay.
- Fine-grained evaluation: Fine-grained linear classification uses resized and center-cropped 224×224 images, with dataset details provided in Table 16.
- Few-shot evaluation: Fine-grained classification evaluates 5-way 5-shot performance across 10 datasets using mean accuracy over 600 randomly sampled tasks.
B Additional Results
Additional evaluations show that StableRep transfers strongly across fine-grained and few-shot image-classification tasks, including comparisons with large-scale real-image pre-training.
- Fine-grained transfer: StableRep trained with synthetic images only approaches OpenAI’s CLIP trained on 400 million real images in fine-grained transfer.
- Fine-grained transfer: Longer StableRep training further improves transferability, while RedCaps pre-training gives StableRep the best average fine-grained accuracy.
- Few-shot classification: StableRep stands out on the majority of evaluated few-shot image-classification datasets, with 95% confidence intervals reported.
C.1 Implementation details
The implementation uses Stable Diffusion v1.5 to generate 512×512 images, stores resized 256×256 versions, and relies on substantial GPU resources for synthesis and training.
- Synthesis: Stable Diffusion v1.5 generates images with 50 DDIM steps, requiring approximately 0.8 seconds per image on an A100 and 2.3 seconds on a V100.
- Image resolution: Synthetic images are generated at 512×512 resolution and resized to 256×256 before storage for comparison with real images.
- Synthesis: Generating ten million images takes approximately 13 hours using 512 V100 GPUs, making image synthesis the slowest pipeline stage.
- Pre-training: StableRep ViT-B/16 pre-training uses four nodes with eight A100 GPUs each and takes approximately 20–23 hours for 35 epochs.
- Examples: Figure 8 shows synthetic examples for four prompts across guidance scales alongside the original real image.