PaperlayerView exact revision on arXiv

Source-linked AI summary

Fake it till you make it: Learning transferable representations from synthetic ImageNet clones

Mert Bulent Sariyildiz, Karteek Alahari, Diane Larlus, Yannis Kalantidis

arXiv:2212.08420v2cs.CVcs.LG

TL;DR

The paper investigates whether real images are necessary for ImageNet classification by replacing them with Stable Diffusion images generated from class names. Using minimal, class-agnostic prompt engineering, it finds strong synthetic-data performance, including transfer results on par with models trained on real data, while acknowledging bias and diversity limitations.

  • Problem

    ImageNet’s real-image curation and annotation are costly, motivating the question of whether text-to-image generators can replace real images for training prediction models.

  • Method

    The study generates ImageNet clones with Stable Diffusion from class names, applies class-agnostic prompt modifications, and trains classification models from scratch using only synthetic images.

  • Results

    Synthetic-trained models achieve strong ImageNet accuracy, narrow gaps on robustness evaluations, and perform on par with real-data models across 15 transfer datasets.

  • Takeaways & Limitations

    Synthetic images can support transferable visual representations whose performance rivals models learned with real images in the evaluated settings.

  • Takeaways & Limitations

    The approach inherits unknown training-data provenance, problematic content, bias, memorization, and insufficient diversity from Stable Diffusion, with mitigation remaining incomplete.

Abstract

from arXiv · show

Recent image generation models such as Stable Diffusion have exhibited an impressive ability to generate fairly realistic images starting from a simple text prompt. Could such models render real images obsolete for training image prediction models? In this paper, we answer part of this provocative question by investigating the need for real images when training models for ImageNet classification. Provided only with the class names that have been used to build the dataset, we explore the ability of Stable Diffusion to generate synthetic clones of ImageNet and measure how useful these are for training classification models from scratch. We show that with minimal and class-agnostic prompt engineering, ImageNet clones are able to close a large part of the gap between models produced by synthetic images and models trained with real images, for the several standard classification benchmarks that we consider in this study. More importantly, we show that models trained on synthetic images exhibit strong generalization properties and perform on par with models trained on real data for transfer. Project page: https://europe.naverlabs.com/imagenet-sd/

1. Introduction

The paper asks whether real images remain necessary for training image prediction models, focusing on replacing ImageNet with synthetic images generated from class names. It finds that synthetic-trained models can approach real-data performance across recognition, robustness, and transfer evaluations.

  • ImageNet-1K contains about 1 million annotated images, whose curation and annotation require substantial money and labor.
  • Stable Diffusion generates fairly realistic, compositional images from text prompts, motivating the question of whether real training images are still needed.
  • The study replaces real ImageNet training images with synthetic images generated from ImageNet class names, then evaluates models on real-image recognition, robustness, and transfer tasks.
  • 91.7% top-5 accuracy on ImageNet-100 and 70.3% top-5 accuracy on ImageNet-1K are achieved using synthetic ImageNet clones.
  • Synthetic-trained models reduce the performance gap on domain-shift and adversarial evaluations and perform on par with real-data models across 15 transfer datasets.

2. Related work

The related work spans synthetic data generation, synthetic ImageNet datasets, and knowledge or dataset distillation. This paper differs by using a general-purpose text-to-image generator to create task-specific synthetic training images without real-image access.

  • Prior synthetic-data approaches often rely on 3D models, renderers, or simulators for annotation-heavy vision tasks.
  • Earlier synthetic ImageNet studies generated class images with conditional GANs such as BigGAN.
  • Knowledge distillation transfers information from a pretrained teacher to a student and usually requires images.
  • This approach can be viewed as image-free distillation: prompting a generic text-to-image model to produce images for training a classifier.
  • Unlike dataset distillation, the method samples from a task-agnostic generator rather than tailoring image synthesis to a specific task.

3. Preliminaries

The task replaces a fixed real-image training set with a text-to-image generator, while retaining standard supervised optimization and evaluating both classification and transferable visual representations.

  • Given class names C and a generator G, the method learns an encoder z = fθ(x) and classifier y = q(z) using only generated images.
  • The encoder and classifier are trained jointly, then evaluated on real images for the original classification task.
  • The learned visual encoder is also evaluated in transfer-learning scenarios beyond the original classification task.
  • Stable Diffusion generates an image x = g(p) from a textual prompt through text conditioning, latent diffusion, and decoding.
  • The default generation settings are 50 diffusion steps and guidance scale 7.5, with guidance scale controlling visual diversity.

4. Generating synthetic ImageNet clones

ImageNet-SD clones are generated from class names and improved with generic prompt modifications that address semantic ambiguity and limited diversity. Lower guidance and varied backgrounds further diversify outputs, although synthetic images remain less varied than real ImageNet images.

  • 4. Generating synthetic ImageNet clones: ImageNet-SD denotes synthetic ImageNet datasets created by Stable Diffusion for the classes of ImageNet.
  • 4.1. Generating datasets using class names: Using synset lemmas as prompts produces photo-realistic images, but some classes exhibit semantic errors, low diversity, or shifts toward sketches and art.
  • 4.2. Addressing issues with semantics and domain: Polysemy causes class-name prompts to produce the wrong visual meaning, as with “papillon” images depicting butterflies instead of the ImageNet dog breed.
  • 4.2. Addressing issues with semantics and domain: Appending WordNet hypernyms or definitions to class names disambiguates problematic classes and can reduce visual-domain mismatch.
  • 4.3. Increasing the diversity of generated images: Real ImageNet images still contain significantly more diversity, settings, backgrounds, and multiple instances than the synthetic outputs.
  • 4.3. Increasing the diversity of generated images: Reducing guidance scale to 2.0 increases diversity despite slightly reducing visual quality, while scene backgrounds provide a class-agnostic diversity strategy.

5. Experiments

Experiments evaluate synthetic ImageNet clones using Stable Diffusion, testing classification on real ImageNet images, distribution shifts, and transfer datasets. Prompt design and dataset scale substantially improve results, while synthetic representations generalize competitively in transfer despite weaker ImageNet-1K classification.

  • Experimental setup: Experiments primarily use ImageNet-100 to compare synthetic-dataset variants and design choices, while preserving ImageNet’s fine-grained classification characteristics.ImageNet-100 is a randomly chosen 100-class subset of ImageNet-1K.
  • Experimental setup: Stable Diffusion generates class-conditioned images, and models are evaluated on real images using either the learned classifier or a transferred linear classifier.Transfer evaluation uses the pretrained encoder as a feature extractor and trains separate linear classifiers.
  • Prompting and scale: Over 50% Top-1 and 76% Top-5 accuracy on ImageNet-100 is achieved by prompting objects on diverse backgrounds at the default guidance scale.Class names alone already exceed 70% Top-5 and reach 43% Top-1 accuracy; hypernyms and definitions improve performance further.
  • Prompting and scale: 73.3% Top-1 and 91.7% Top-5 accuracy on ImageNet-100 are reached with 50× synthetic images, yielding gains of up to 8.5% in Top-1 accuracy.Lower guidance scale improves overall performance by producing more diverse image sets.
  • Robustness: Synthetic ImageNet-100 models outperform real-image models on ImageNet-Sketch and ImageNet-R, while the best 50× model rivals the real-data baseline on ImageNet-A.The same pattern does not hold for ImageNet-1K, where the synthetic model lags behind real-data models on all reported comparisons.
  • Transfer learning: Representations learned purely from synthetic images show transfer generalization comparable to representations trained on thousands or millions of real images.The best transfer protocol reaches 70.4% Top-1 accuracy on ImageNet-1K, and 10× more synthetic images already surpasses the real-data model on average transfer performance.

6. Discussion

The discussion highlights synthetic data’s scalability and broad applicability, alongside limits imposed by generator capability, compute, moderation, and inherited bias. Stable Diffusion’s unknown and potentially problematic training data, architectural constraints, limited diversity, and stereotype reinforcement remain unresolved concerns.

  • Applicability beyond ImageNet: ImageNet-SD can extend beyond ImageNet because it requires only class names and short textual descriptions to disambiguate semantics.Such descriptions may be acquired semi-automatically from sources such as Wikipedia.
  • Scaling laws for synthetic data: Synthetic-data scaling is bounded by the generator’s expressivity and by computation, storage, and content-moderation costs.The promise of an effectively unbounded dataset therefore does not eliminate practical resource and governance constraints.
  • Data and model bias: Outsourcing curation and annotation to a text-to-image generator shifts responsibility to training data whose provenance may be unknown.The study uses Stable Diffusion, trained on automatically filtered web-scraped LAION-2B data that has been reported to contain problematic content.
  • Data and model bias: Generator architecture can propagate and potentially amplify existing bias, including limited diversity and reinforced stereotypes.The explored prompt changes mitigate this issue only to a limited extent, and the authors call for broader multidisciplinary discussion of societal implications.

7. Conclusions

The paper studies whether ImageNet can be replaced by synthetic images and finds strong classification and transfer performance from models trained without real training images. It presents this as an initial demonstration of synthetic visual representation learning and suggests combining synthetic and real data for adaptation.

  • Conclusions: The study asks to what extent ImageNet can be replaced by a dataset synthesized by a text-to-image generator.
  • Conclusions: Models trained on synthetic data exhibit generalization capability that rivals models learned with real images.
  • Conclusions: The authors frame these results as a first glimpse of synthetic visual representation learning and envision using synthetic datasets alongside real ones for fine-tuning or adaptation.

A. Implementation details

Implementation uses ResNet50 encoders trained with a DINO-style multi-crop augmentation pipeline and standard SGD optimization. Stable Diffusion generates 512 × 384 RGB images using 50 diffusion steps and guidance scale 7.5.

  • Training configuration: All experiments use a ResNet50 encoder trained for 100 epochs with mixed precision on four GPUs and synchronized batch normalization.Optimization uses SGD with momentum 0.9, batch size 256, warmup over the first 10% of iterations, and cosine decay.
  • Training configuration: The default augmentation pipeline uses one global crop and eight local crops from DINO-style multi-crop augmentation.
  • Image generation: Stable Diffusion uses 50 diffusion steps and guidance scale 7.5 to generate RGB images at 512 × 384 resolution.

B. Evaluation protocol

The evaluation uses pretrained encoders and classifiers to test synthetic-data models on real-image benchmarks, domain shifts, adversarial settings, and transfer datasets. Linear classifiers are trained on frozen representations for novel-class transfer tasks.

  • Models are evaluated on ImageNet test sets using pretrained encoders and classifiers learned during synthetic-image pretraining.
  • Novel-class transfer evaluations freeze each pretrained encoder and train a new classifier from scratch.
  • Transfer learning follows linear-classification protocols by extracting image representations and fitting logistic-regression classifiers.
  • Data augmentation: Strong augmentations increase ImageNet-100-SD performance by over 14%, while improving real-image models by less than one percent.

C.2. Results on the ImageNet-CoG [78] benchmark

The ImageNet-CoG benchmark tests concept generalization across five ordered levels whose concepts become increasingly semantically distant from ImageNet-1K. Synthetic-image representations are comparable to real-image representations at the most challenging level, while the gap favors real-data models at nearer levels.

  • ImageNet-CoG evaluates ImageNet-1K training classes plus five 1000-concept levels drawn from ImageNet-19K.The levels are ordered from concepts closer to ImageNet-1K to concepts that are progressively more semantically distant.
  • At L5, representations learned from synthetic images perform comparably to representations learned from real images.L5 is identified as the most challenging concept-generalization level.
  • Toward L1, the performance gap increases in favor of RSB-A1.
  • 70.4% Top-1 accuracy is reached after training classifiers only with real ImageNet-1K images, significantly narrowing the gap to RSB-A1.
  • The broader evaluation compares real- and synthetic-data models across ImageNet test sets, small-scale and long-tail transfer datasets, and CoG levels.

D.1. Semantic errors

Synthetic ImageNet images exhibit several semantic and visual fidelity problems, including polysemy errors, biodiversity misrepresentation, object omission, and a diversity–semantic accuracy trade-off.

  • Semantic errors: Polysemous class names produce wrong semantics, including cars for “green mamba” and non-insect objects for “walking stick”.Appending hypernyms or definitions fixes many cases but can also worsen results, such as “shih-tzu” becoming toy dogs and “boathouse” becoming sheds.
  • Semantic errors: Hypernym-based prompt corrections are inconsistent: they resolve many polysemy issues but introduce errors for “shih-tzu” and “boathouse”.The “toy dog” hypernym leads to dog-shaped toys, while “shed” removes the defining water setting from boathouse images.
  • Misrepresentation of biodiversity: Synthetic images highly misrepresent biodiversity, often capturing the coarse class “crab” without the enlarged claw distinguishing fiddler crabs.The generated crab diversity can exceed a single species, and the authors report this issue across many fine-grained animal and fungi domains.
  • Semantic errors: Adding diverse backgrounds sometimes causes images to omit the target object and represent only the surrounding scene.Some backgrounds nevertheless retain traces of the object’s shape or texture, such as a pedestal-like table or green shirts.
  • Semantic errors: Lower guidance scales or added backgrounds partially improve diversity, but greater diversity correlates with more semantic errors.Class-name-only prompts often lack diversity, while the most diverse synthetic dataset shows semantic issues more frequently.
  • Non-natural images: A non-negligible fraction of generated images are non-natural illustrations, graphics, or paintings, which may support robustness to related domain changes.Stable Diffusion generation quality also depends on guidance scale and diffusion steps: 25–50 steps produce sharp images, while scales above 10 become hyperrealistic.
Loading 2212.08420v2…