PaperlayerView exact revision on arXiv

Source-linked AI summary

Is synthetic data from generative models ready for image recognition?

Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, Xiaojuan Qi

arXiv:2210.07574v2cs.CV

TL;DR

The paper asks whether high-fidelity synthetic images from text-to-image models are applicable to image recognition, especially when real labeled data are scarce or for transfer-learning pre-training. It studies GLIDE-based synthetic data with CLIP across zero-shot, few-shot, and pre-training settings, finding benefits in all three while documenting important limits and practical strategies.

  • Problem

    The applicability of high-fidelity text-to-image synthetic images to recognition tasks remains under-explored despite the cost, time, privacy, and usage-rights concerns of collecting large labeled datasets.

  • Method

    The study evaluates GLIDE-generated synthetic data with frozen-feature CLIP-based classifier tuning for zero-shot and few-shot recognition, and examines synthetic data for large-scale transfer-learning pre-training.

  • Results

    Synthetic data improve zero-shot and few-shot classification and provide strong pre-training performance, including results that outperform ImageNet pre-training.

  • Takeaways & Limitations

    Synthetic data are a promising resource for data-scarce recognition and model pre-training, especially when generation strategies address diversity, noise, reliability, and domain gaps.

  • Takeaways & Limitations

    Synthetic data remain less effective than real data for training classification models, and the study could not further scale data and model size because of limited computational resources.

Abstract

from arXiv · show

Recent text-to-image generation models have shown promising results in generating high-fidelity photo-realistic images. Though the results are astonishing to human eyes, how applicable these generated images are for recognition tasks remains under-explored. In this work, we extensively study whether and how synthetic images generated from state-of-the-art text-to-image generation models can be used for image recognition tasks, and focus on two perspectives: synthetic data for improving classification models in data-scarce settings (i.e. zero-shot and few-shot), and synthetic data for large-scale model pre-training for transfer learning. We showcase the powerfulness and shortcomings of synthetic data from existing generative models, and propose strategies for better applying synthetic data for recognition tasks. Code: https://github.com/CVMI-Lab/SyntheticData.

1 INTRODUCTION

This paper studies whether text-to-image synthetic data can support image recognition in data-scarce classification and large-scale transfer-learning pre-training. It finds benefits across zero-shot, few-shot, and pre-training settings, while identifying strategies and limitations.

  • Prior work: Earlier synthetic-data studies used GANs for classification, segmentation, or contrastive learning, but generally addressed smaller-scale tasks or narrower settings.The cited approaches include BigGAN, StyleGAN, and GAN latent-space manipulation.
  • Study scope: The study presents the first investigation of state-of-the-art text-to-image models for image recognition, using GLIDE to examine classifier improvement and transfer-learning pre-training.Classifier improvement is studied only in zero-shot and few-shot settings, where the positive impact of synthetic data is expected to be more relevant.
  • Zero-shot classification: Synthetic data improves zero-shot classification across 17 diverse datasets, with average top-1 accuracy increasing by 4.31% and gains reaching 17.86% on EuroSAT.The study designs diversified text prompts and uses CLIP feature correlations to increase diversity, reduce noise, and enhance reliability.
  • Few-shot classification: In few-shot classification, synthetic data are beneficial and achieve a new state of the art, though their impact is less significant than in zero-shot learning.The paper identifies the domain gap between synthetic and downstream data as a challenge and uses real images to guide generation.
  • Transfer-learning pre-training: For large-scale transfer-learning pre-training, synthetic data outperform ImageNet pre-training and work especially well for unsupervised pre-training with ViT-based backbones.Increasing label-space size expands data amount and diversity, while combining synthetic data with ImageNet initialization further improves performance.

2 RELATED WORKS

Synthetic data for image recognition includes traditional simulation-generated datasets and generative-model images. Text-to-image diffusion models offer high-quality conditional synthesis that broadens the potential uses of generative synthetic data.

  • Synthetic data for image recognition mainly takes the form of traditional simulation-generated datasets or images produced by generative models.
  • Traditional synthetic-data pipelines can differ from real-world data and require substantial physical storage and generation resources.
  • Generative models can produce photorealistic images closer to real data, require less storage, and potentially provide an unlimited data supply.
  • Diffusion models learn to reverse a noising process that models the underlying data distribution, enabling novel images to be sampled from a Gaussian prior.
  • Text-to-image generation conditions diffusion-based sampling on natural-language descriptions, and models including Stable Diffusion, DALL-E2, Imagen, and GLIDE achieve high synthesis quality.

3 IS SYNTHETIC DATA READY FOR IMAGE RECOGNITION?

The study finds that synthetic images from text-to-image models can substantially benefit zero-shot and few-shot recognition and can support transfer-learning pre-training, but their effectiveness depends on diversity, reliability, domain alignment, and access to strong initialization or real-data guidance.

  • Zero-shot image recognition: 4.31% average top-1 accuracy gain for CLIP-RN50 and 2.90% for CLIP-ViT-B/16 across 17 zero-shot datasets, with a 17.86% gain on EuroSAT.Performance varies with GLIDE’s training distribution and the resulting domain gaps across datasets.
  • Zero-shot image recognition: Language enhancement increases synthetic-data diversity, while CLIP-based filtering improves reliability by removing low-confidence samples.Language enhancement adds gains on most cases but can introduce noise and reduce performance without filtering.
  • Zero-shot image recognition: Classifier-only tuning performs best for CLIP, whereas tuning the pre-trained encoder degrades performance, primarily because of domain gaps.The default setting tunes only the classifier, corresponding to 0.02% of full-model parameters in the EuroSAT study.
  • Limitations and scope: Synthetic data are inadequate for training classifiers from scratch, remaining less data-efficient and less effective than real data.The paper’s overall conclusion therefore favors synthetic data for settings using strong pre-trained models or real-data guidance rather than standalone scratch training.
  • Few-shot image recognition: Synthetic data improves few-shot classification and reaches state-of-the-art performance, but its benefit diminishes as more real-data shots become available.Real images have no synthetic-to-real domain gap, making each real sample more effective than a synthetic sample.
  • Few-shot image recognition: Real-data guidance reduces domain differences in few-shot learning, while mix training and frozen batch normalization further improve classifier learning.Mix training lets real and synthetic data regularize one another; freezing BN avoids unreliable statistics and cross-domain statistical differences.
  • Large-scale pre-training: Synthetic pre-training approaches ImageNet-1K pre-training and can surpass it as synthetic data size and label-space diversity increase.With 2.4M synthetic images, MoCo v2 reaches 82.13%; with IN-2K Syn, it reaches 82.29%, versus 82.44% for the IN-1K Real counterpart.

4 CONCLUSION

The study finds that synthetic data from text-to-image models benefit zero-shot and few-shot classifier learning and show strong potential for transfer-learning pre-training. It also identifies scaling, language enhancement, and model-capability limitations for further investigation.

  • Synthetic data bring significant performance boosts for zero-shot and few-shot recognition, yielding new state-of-the-art performance.
  • Synthetic data show strong potential for model pre-training, even surpassing standard ImageNet pre-training.
  • Larger synthetic data amounts and label-space diversity improve performance, but the study could not investigate larger scales, model sizes, or advanced architectures.

A.1 LIMITATIONS AND FUTURE DIRECTIONS

The paper outlines future work on language enhancement, larger synthetic datasets and models, and joint tuning of generators with downstream predictors. These directions address noisy data, limited scale, and domain adaptation.

  • Language enhancement enlarges synthetic-data diversity but can generate substantial noise, motivating effective constraints.
  • Zero-shot and few-shot studies used median-sized synthetic datasets because of limited computational resources, leaving larger data amounts and diversity for future work.
  • Jointly tuning the image generator with downstream predictors could generate in-domain images and potentially support domain generalization.
  • Scaling synthetic data, label-space diversity, model size, and model capability remains unexplored because training larger models may take months.

A.2 ANALYSIS OF DIFFERENCES OF PERFORMANCE BOOST ON DIFFERENT DATASETS IN ZERO-SHOT SETTINGS

Performance boosts vary across datasets because synthesized-image quality, domain gaps, and task difficulty differ. The paper suggests tuning the generator with the downstream model to produce more in-domain images and reduce domain gaps.

  • Dataset-specific gains reflect synthesized-image quality, domain gaps, and task difficulty.DTD gains were 0.96%, SUN397 gains were 1.56%, and ImageNet gains were 0.45%.
  • The generator’s training-data distribution can create different domain gaps across datasets, making quantitative relationships between data and performance an open problem.
  • Tuning the image generator with the downstream model is proposed as a way to generate in-domain images and better reduce downstream domain gaps.

B.1 MAIN RESULTS ON FEW-SHOT TASKS

Synthetic data boosts few-shot learning across the evaluated datasets and achieves new state-of-the-art performance.

  • Synthetic data boosts few-shot learning performance across all 8 evaluated datasets and achieves new state-of-the-art performance.

B.2 REAL-WORLD DATA WITH DOMAIN SHIFT

Domain gaps constrain how synthetic data affects pretrained encoders, while label-space alignment and real-data guidance offer ways to address these shifts. Increasing synthetic data eventually yields diminishing returns as redundancy grows.

  • REAL-WORLD DATA WITH DOMAIN SHIFT: Domain-shifted real data behave similarly to synthetic data: end-to-end tuning slightly harms the pretrained encoder, unlike in-domain data.These results suggest that domain gap is the main reason for the degradation.
  • REAL-WORLD DATA WITH DOMAIN SHIFT: Synthetic data can tailor the label space to downstream tasks and use real samples to guide synthesis, potentially reducing category shifts.
  • REAL-WORLD DATA WITH DOMAIN SHIFT: Beyond a certain amount, additional synthetic training data provide little performance gain because diversity does not scale proportionally and redundant samples appear.

B.4 CLAIFICATION OF ZERO-SHOT CLASSIFICATION PERFORMANCE DIFFERENCE

The reported zero-shot baseline differs from the original CLIP result because the study uses simple prompts rather than dataset-specific prompt ensembles. FID provides a separate measure of synthetic-image distance and quality, while generation strategies trade diversity against class fidelity.

  • CLAIFICATION OF ZERO-SHOT CLASSIFICATION PERFORMANCE DIFFERENCE: Using simple prompts lowers the baseline relative to the original CLIP paper, which used specifically designed prompt ensembles for each dataset.
  • CLAIFICATION OF ZERO-SHOT CLASSIFICATION PERFORMANCE DIFFERENCE: 56.33% average performance on 13 datasets with CLIP’s prompt ensemble approaches the original reported 57.03%.The study’s average increased from 56.14% to 56.33% after matching the prompt design.
  • CLAIFICATION OF ZERO-SHOT CLASSIFICATION PERFORMANCE DIFFERENCE: FID measures distance between real and generated image features, with lower scores associated with higher image quality.
  • CLAIFICATION OF ZERO-SHOT CLASSIFICATION PERFORMANCE DIFFERENCE: Language enhancement increases diversity but can hurt class fidelity, while combining it with CLIP filtering further reduces FID and improves class fidelity.
  • CLAIFICATION OF ZERO-SHOT CLASSIFICATION PERFORMANCE DIFFERENCE: Language enhancement may shift attention toward introduced objects, producing images that omit the desired class.

B.8 VISUALIZATION: SYNTHETIC DATA IN FEW-SHOT SETTINGS

In few-shot settings, real filtering and real guidance reduce the domain gap between synthesized and target-domain images, with real guidance producing the closest visual match.

  • Real filtering and real guidance both reduce the domain gap from real target-domain images, while real guidance approaches them more closely than real filtering.

B.9 VISUALIZATION: SYNTHETIC DATA FOR DIFFERENT DATASETS

GLIDE-generated images are generally high quality across the visualized datasets, but quality varies by task, with some low-quality examples. The paper also describes diffusion-based generation and guidance strategies used to produce and refine synthetic data.

  • Visualization: GLIDE produces high-quality synthesized images for most visualized datasets, although many unsatisfactory examples occur for DTD.The images are randomly selected, and each row contains images from the same class.
  • Visualization: The authors attribute task-specific low-quality generation primarily to a domain gap between the generator’s training data and the recognition task.They expect future text-to-image models to improve image quality and suggest lower-quality images may still help pre-training.
  • Generation method: Text-to-image diffusion models condition image generation on a natural-language description, extending unconditional diffusion from q(x0) to q(x0 | c).The conditional formulation requires the sampled image to match the given description.
  • Generation method: GLIDE uses a coarse-to-fine two-stage framework with classifier-free guidance, trading off sampling quality and diversity through guidance scale s.The reported default is s = 3, with T = 100 in the coarse stage and T = 27 in the upsampler stage.
  • Few-shot guidance: For few-shot generation, Real Guidance initializes denoising from a noised in-domain reference image rather than pure noise, helping generated images share in-domain properties.The strategy starts at time-step t⋆ and uses different empirically selected values for different shot counts.
  • Data generation: Synthetic-data generation uses class-based prompts or diversified sentences produced by a word-to-sentence T5 model, followed by GLIDE image synthesis.The described experiments generate synthetic images using the official GLIDE hyperparameters and use 2000 synthetic images per class in the stated setup.

C.5.3 PRE-TRAINING SETTING

The pre-training experiments vary label-space specificity, synthetic-data scale, backbone, initialization, and supervision while keeping transfer evaluation protocols defined for downstream tasks.

  • Data construction: Downstream-aware pre-training uses prompts derived from CIFAR-100 labels and synthetic datasets of 1.2M, 2.4M, or 3.6M images.Downstream-agnostic generation instead uses ImageNet-1K or ImageNet-2K label spaces with 1.2M, 2.4M, or 4M images.
  • Downstream-aware pre-training: Downstream-aware training uses AdamW, weight decay 0.9, batch size 512, 90 epochs, and cosine learning-rate annealing.The initial learning rate is 1e-4 from random initialization and 1e-5 from ImageNet-pretrained weights.
  • Downstream-agnostic pre-training: Downstream-agnostic synthetic supervised pre-training with ResNet50 uses SGD, batch size 512, weight decay 1e-4, and learning-rate drops every 30 epochs.Training lasts 90 epochs, with initial learning rates of 0.2 from random initialization and 0.001 from ImageNet-pretrained weights.
  • Downstream-agnostic pre-training: For DeiT-S, the study replaces ImageNet pre-training data with synthetic data while leaving the official training hyperparameters unchanged.This isolates the effect of replacing the pre-training dataset within the referenced training scripts.
  • Self-supervised pre-training: Downstream-agnostic synthetic self-supervised pre-training uses MoCo v2 hyperparameters, with an initial learning rate of 0.003 when initialized from ImageNet weights.Training from random initialization follows the original implementation’s hyperparameters.
  • Transfer evaluation: Transfer evaluation for object detection on PASCAL VOC 2012 uses Faster R-CNN with backbones initialized by the pre-trained weights.The evaluation setup follows the protocols used in MoCo.
Loading 2210.07574v2…