Source-linked AI summary
Synthetic Data from Diffusion Models Improves ImageNet Classification
Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, David J. Fleet
TL;DR
The paper asks whether current diffusion models can generate useful data for challenging ImageNet classification. It fine-tunes a large text-to-image model into class-conditional generators and finds strong generative metrics, classification scores, and augmentation gains, with high-resolution scaling showing a limitation at very large synthetic-set sizes.
Problem
The paper asks whether diffusion models can produce synthetic ImageNet data of sufficient quality and diversity to improve challenging classification tasks.
Method
The authors fine-tune a large pretrained text-to-image diffusion model on ImageNet and evaluate its samples for generation, CAS, and classifier augmentation.
Results
Synthetic-data augmentation improves ImageNet classification across ResNet and Transformer models, while the generator achieves FID 1.76 and IS 239 at 256×256 and CAS 69.24 at 1024×1024.
Takeaways & Limitations
Fine-tuned text-to-image diffusion models can provide effective class-conditional synthetic data for ImageNet classification augmentation.
Takeaways & Limitations
At high resolutions, classification performance drops for synthetic datasets larger than 1M images, possibly indicating generative-model bias and a need for more sophisticated training methods.
Abstract
from arXiv · showhide
Deep generative models are becoming increasingly powerful, now generating diverse high fidelity photo-realistic samples given text prompts. Have they reached the point where models of natural images can be used for generative data augmentation, helping to improve challenging discriminative tasks? We show that large-scale text-to image diffusion models can be fine-tuned to produce class conditional models with SOTA FID (1.76 at 256x256 resolution) and Inception Score (239 at 256x256). The model also yields a new SOTA in Classification Accuracy Scores (64.96 for 256x256 generative samples, improving to 69.24 for 1024x1024 samples). Augmenting the ImageNet training set with samples from the resulting models yields significant improvements in ImageNet classification accuracy over strong ResNet and Vision Transformer baselines.
1. Introduction
The paper asks whether diffusion models can generate useful data for challenging discriminative tasks, then shows that fine-tuned text-to-image models achieve strong ImageNet generation and classification results.
- Diffusion models are evaluated as sources of synthetic data for generative augmentation on the highly tuned ImageNet classification benchmark.
- Large-scale text-to-image models can be fine-tuned into state-of-the-art class-conditional ImageNet generators.
- FID 1.76 and IS 239 are achieved for 256 × 256 generated samples.
- Fine-tuned models establish new state-of-the-art Classification Accuracy Scores when ResNet-50 classifiers trained on synthetic data are evaluated on real ImageNet data.
- Combining synthetic and real data improves classification accuracy across convolutional and Transformer-based architectures.
2. Related Work
Related work covers synthetic data generation for vision tasks, diffusion-model applications, and early uses of text-to-image diffusion models for classification augmentation.
- Prior synthetic-data research includes rendered, simulated, and data-driven approaches for segmentation, optical flow, human motion, and other dense prediction tasks.
- GANs have been the predominant data-driven approach to generating synthetic natural images, while recent work has begun using text-to-image diffusion models.
- The paper’s diffusion-model setup transfers knowledge from a pretrained multimodal generator fine-tuned on ImageNet to a classification model through synthetic data.
- Diffusion models have been applied to image, speech, and video generation, as well as image colorization, super-resolution, inpainting, and semantic editing.
- Earlier studies used GLIDE or pretrained diffusion models for zero-shot, few-shot, CIFAR-100, and individual-image augmentation settings.
3. Background
The background explains diffusion sampling and motivates Classification Accuracy Score as a downstream-oriented alternative to standard generative metrics.
- Diffusion models add Gaussian noise progressively and learn a reverse process that refines random noise into an image.
- Diffusion research has introduced cascaded resolutions, noise schedules, guidance mechanisms, distillation, and alternative denoising objectives.
- FID and Inception Score are standard visual-quality metrics, but their relationship to downstream classification accuracy is inconsistent.
- Classification Accuracy Score measures ImageNet validation accuracy of a ResNet-50 trained solely on generated samples.
- Before this work, synthetic-only models underperformed real-data models, and adding modest synthetic data could reduce performance.
4. Generative Model Training and Sampling
The study fine-tunes a large pretrained Imagen cascade on ImageNet and systematically tunes sampling parameters using FID, IS, and CAS across resolutions.
- 4. Generative Model Training and Sampling: The experiments test whether large-scale text-to-image models can become class-conditional ImageNet generators and support generative data augmentation.
- 4. Generative Model Training and Sampling: Imagen is used as a pretrained foundation and fine-tuned with ImageNet data to address alignment between generated images and ImageNet classes.
- 4. Generative Model Training and Sampling: Only the 64×64 base and 64×64→256×256 super-resolution models are fine-tuned, while the final super-resolution module and text encoder remain unchanged.
- 4.2. Sampling Parameters: Sampling studies vary guidance weight, log-variance, denoising steps, and noise conditioning augmentation to select parameters for downstream classification.
- 4.2. Sampling Parameters: Smaller guidance weights improve CAS but reduce Inception Score, while larger noise augmentation increases diversity and FID.
- 4. Generative Model Training and Sampling: Synthetic datasets are constructed with balanced ImageNet classes and sizes from 1.2M to 12M images.
5. Results
Fine-tuned Imagen diffusion models achieve strong ImageNet generative quality and classification-transfer performance, while synthetic images substantially improve classifiers when combined with real data. Gains persist as synthetic data scales, especially at 64×64 resolution, though higher-resolution augmentation eventually stops improving.
- Sample Quality: FID and IS: Fine-tuning Imagen produces state-of-the-art ImageNet sample quality without design changes.The approach is evaluated with FID and Inception Score against class-conditional and guidance-based methods.
- Classification Accuracy Score: 69.24% Top-1 CAS is achieved with 1024×1024 generated samples, substantially outperforming previous methods and narrowing the gap with real-data training.The 256×256 model also outperforms prior methods by a good margin on both Top-1 and Top-5 accuracy.
- Classification Accuracy with Different Models: Figure 5 compares class-wise accuracy for real- versus generated-data-trained models across CDM and fine-tuned Imagen resolutions.Fine-tuned Imagen models outperform real-data models on more classes, particularly at 1024×1024, unlike the weaker CDM comparison.
- Classification with Different Models: Fine-tuned diffusion samples improve Top-1 accuracy across multiple ConvNet and Transformer classifiers when augmenting 1.2M real ImageNet images.Classifiers trained solely on generated samples perform worse than those trained on real data, but mixing synthetic and real data provides a substantial boost.
- Merging Real and Synthetic Data at Scale: At 64×64 resolution, classification performance continues improving as synthetic data increases to nine times the real dataset, reaching 12M total images.At 256×256 and 1024×1024, gains remain significant up to four or five times the real training-set size, but do not continue similarly at larger augmentation levels.
6. Conclusion
Large-scale text-to-image diffusion models can be fine-tuned for effective ImageNet generative augmentation, achieving strong generative and classification metrics. Improvements extend across substantial synthetic datasets and multiple model families, although high-resolution training can degrade beyond a dataset-size threshold.
- Fine-tuned diffusion models achieve SOTA FID of 1.76 and Inception Score of 239 at 256×256 resolution.
- Classification Accuracy Score reaches 64.96 for 256×256 models and 69.24 for 1024×1024 generated samples.
- ImageNet classification improvements extend to large amounts of generated data across ResNet and Transformer-based models.
- At high resolutions, performance drops for synthetic datasets larger than 1M images, possibly indicating generative-model bias and a need for more sophisticated training methods.
A.1. Hyper-parameters for Imagen fine-tuning and sample generation.
The appendix examines how diffusion sampling and fine-tuning hyper-parameters jointly affect image quality, diversity, speed, and classification accuracy. It evaluates guidance, noise augmentation, diffusion steps, variance, and super-resolution refinement using FID, Inception Score, and CAS trade-offs.
- Sampling hyper-parameters: Larger numbers of diffusion steps are often associated with higher-quality images and lower FID, while noise-conditioning augmentation affects later cascade stages.Noise-conditioning augmentation adds Gaussian noise to one Imagen cascade stage’s output before the next super-resolution stage receives it.
- Guidance and thresholding: Smaller classifier-free guidance weights favor diversity over image fidelity and avoid requiring dynamic thresholding during inference.The method instead uses a static threshold to clip large pixel values at each denoising step.
- Metric trade-offs: Guidance-weight sweeps expose Pareto trade-offs between classification accuracy score and train FID at 256×256 resolution.The curves vary guidance weight and, in related settings, sampling steps and conditioning noise augmentation.
- Super-resolution refinement: Fine-tuning the super-resolution model jointly improves classification accuracy and FID relative to vanilla Imagen.This comparison is reported for the fine-tuned model’s super-resolution stage.
- Super-resolution refinement: At 1024×2014 resolution, sampling refinement compares classification accuracy with guidance weights and Inception Score under varying noise conditions.The figure evaluates guidance increases from 1.0 to 5.0 for the CAS–IS relationship.
A.2. Class Alignment of Imagen vs. Fine-Tuned Imagen
The appendix compares vanilla Imagen with fine-tuned Imagen under high- and low-guidance sampling strategies. Fine-tuning and lower guidance improve class-label alignment and diversity, while higher guidance can improve photorealism at the expense of diversity.
- Sampling strategies: The comparison samples fine-tuned Imagen with either vanilla Imagen hyper-parameters or the proposed lower-guidance hyper-parameters.The high-guidance strategy uses guidance weights of 10 for the 64×64 base model and 20 and 8 for subsequent super-resolution models.
- Class alignment: Fine-tuning and careful sampling-parameter choices improve alignment between generated images and class labels.The comparison includes vanilla Imagen and fine-tuned Imagen sampled with high or low guidance.
- Quality–diversity trade-off: Lower guidance improves sample diversity, whereas higher guidance can improve photorealism but lessens diversity.This trade-off is illustrated across the 1024×1024 example-image comparisons.
A.3. High Resolution Random Samples from the ImageNet Model
The fine-tuned ImageNet model generates random 1024×1024 samples across a broad set of ImageNet classes. The examples include animals, foods, vehicles, and other objects.
- Class coverage: Random 1024×1024 samples span 18 ImageNet classes, including snail, panda, orange, school bus, macaw, and green mamba.The listed examples also include birds, amphibians, reptiles, mammals, a steam locomotive, and a carved pumpkin.
A.4. Hyper-parameters and model selection for ImageNet classifiers.
The section specifies training hyperparameters for the CAS ResNet-50 classifier and the ResNet, ResNet-RS, and Transformer classifiers used for ImageNet accuracy reporting. It also describes the training-data regimes and summarizes ConvNet and vision-transformer settings in separate tables.
- Training hyperparameters cover the CAS ResNet-based model and the ResNet, ResNet-RS, and Transformer models used for classifier accuracy reporting.
- The CAS ResNet-50 classifier uses single-crop evaluation, SGD with momentum, 90 epochs, and a learning rate schedule with warmup and drops at epochs 30, 60, and 80.The learning rate increases linearly from 0.0 to 0.4 over five epochs, then decreases by a factor of 10 at each specified epoch.
- Vision-transformer models are trained from scratch to convergence using real-only, real-plus-generated, or generated-only data.The training recipes mainly follow prior recipes for ViT-S/16 and DeiT models.
- Table A.1 lists hyperparameters for ConvNet architectures, including ResNet, ResNet-RS, and the CAS ResNet-50 classifier.
- Table A.2 lists hyperparameters for ViT-S/16 and DeiT-S, DeiT-B, and DeiT-L architectures.