PaperlayerView exact revision on arXiv

Source-linked AI summary

Leaving Reality to Imagination: Robust Classification via Generated Datasets

Hritik Bansal, Aditya Grover

arXiv:2302.02503v2cs.CVcs.AIcs.LGcs.MM

TL;DR

Image classifiers can lose substantial performance on naturally shifted data, and the paper asks whether generated datasets can help address that gap. It trains classifiers with real data augmented by Stable Diffusion generations and finds robust performance gains on natural shifts, while identifying scope and evaluation limitations.

  • Problem

    Image classifiers show performance gaps between i.i.d. test data and naturally shifted data, motivating study of whether generated datasets influence natural robustness.

    Source: §1 · pp. 1–2

  • Method

    The paper generates Stable Diffusion images for ImageNet classes and trains classifiers on real data augmented with generated data, without finetuning the base generative model on real images.

    Source: §1 · p. 2, §2.4 · p. 4

  • Results

    Real-plus-generated training achieves high absolute performance and effective robustness on natural shifts; base-model generations yield higher shift accuracy than finetuned-model generations, while standard classifiers show up to 20% absolute degradation on generated images.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Generated data from the base generative model has greater practical usefulness for training robust classifiers than data from a generative model finetuned on real ImageNet data.

    Source: §1 · p. 2, §7 · p. 10

    The evaluation focuses on ImageNet-1K objects and robustness, leaving fairness, privacy, and generalization to larger datasets such as ImageNet-21K unexplored.

    Source: A Limitations · p. 16

Abstract

from arXiv · show

Recent research on robustness has revealed significant performance gaps between neural image classifiers trained on datasets that are similar to the test set, and those that are from a naturally shifted distribution, such as sketches, paintings, and animations of the object categories observed during training. Prior work focuses on reducing this gap by designing engineered augmentations of training data or through unsupervised pretraining of a single large model on massive in-the-wild training datasets scraped from the Internet. However, the notion of a dataset is also undergoing a paradigm shift in recent years. With drastic improvements in the quality, ease-of-use, and access to modern generative models, generated data is pervading the web. In this light, we study the question: How do these generated datasets influence the natural robustness of image classifiers? We find that Imagenet classifiers trained on real data augmented with generated data achieve higher accuracy and effective robustness than standard training and popular augmentation strategies in the presence of natural distribution shifts. We analyze various factors influencing these results, including the choice of conditioning strategies and the amount of generated data. Additionally, we find that the standard ImageNet classifiers suffer a performance degradation of upto 20\% on the generated data, indicating their fragility at accurately classifying the objects under novel variations. Lastly, we demonstrate that the image classifiers, which have been trained on real data augmented with generated data from the base generative model, exhibit greater resilience to natural distribution shifts compared to the classifiers trained on real data augmented with generated data from the finetuned generative model on the real data. The code, models, and datasets are available at https://github.com/Hritikbansal/generative-robustness.

1 Introduction

The paper asks whether modern generated datasets can improve classifiers’ natural robustness amid performance gaps on naturally shifted data. It finds benefits from mixing real and generated data, while also examining generation strategies and classifier performance on generated images.

1 Introduction

  • The work studies the gap between classifiers’ performance on i.i.d. test data and naturally shifted data, which limits applications such as autonomous driving and medical diagnosis.A ResNet-101 trained on ImageNet-1K experiences a 50% performance reduction on ImageNet-Sketch.Source: §1 · p. 1
  • In-the-wild generative models can produce varied data through repeated queries and conditioning strategies, unlike a fixed set of hand-engineered augmentations.The paper reports that substituting these generations with hand-crafted augmentations or traditional class-conditional generative-model outputs is less effective.Source: §1 · p. 2
  • The study asks how datasets from modern in-the-wild generative models influence natural robustness, using Stable Diffusion to generate 1.3M training and 50K validation images.Generation uses ImageNet-1K and ImageNet-100 classes, without finetuning the base generative model on real images.Source: §1 · p. 2
  • Real-data augmentation with generated data can yield high absolute performance and effective robustness on natural distribution shifts, while removing either data source reduces a corresponding outcome.The authors also report that this mixture does not tailor its generated dataset to a single target domain.Source: §1 · p. 2
  • Classifiers including ResNet-101, finetuned CLIP, and Vision Transformers suffer an absolute 20% degradation on generated images prompted with class labels.The authors interpret this as fragility to newly generated natural variations.Source: §1 · p. 2
  • Real data augmented with base-model generations achieves higher natural-shift accuracy than real data augmented with generations from a model finetuned on real ImageNet data.The base and finetuned generated datasets are publicly available.Source: §1 · p. 2

2 Background

The background establishes supervised classification and robustness measures, then describes generative modeling and Stable Diffusion conditioning strategies used to create training data.

2.1 Supervised Classification

  • Supervised classification trains a model to represent P(y|x) from labeled data and evaluates it on a held-out test set using accuracy.The training setup assumes i.i.d. samples and uses empirical risk minimization.Source: §2.1 · pp. 2–3

2.2 Robustness

  • Accuracy gap compares shifted-set accuracy with source-test accuracy, while effective robustness measures shifted accuracy relative to a fitted standard-classifier baseline.Positive effective robustness indicates improvement over standard training.Source: §2.2 · p. 3

2.3 Generative Modeling

  • Generative models learn a data distribution and can sample new images from a class-conditional distribution given a conditioning variable.The paper denotes this stochastic mapping as G.Source: §2.3 · p. 3

2.4 Data Generation using Stable Diffusion

  • Stable Diffusion primarily generates images from class-label text templates, while the study also considers source-image conditioning and combined image-and-label conditioning.The work evaluates generated data for quality, consistency, and diversity through human and automatic assessments.Source: §2.4 · p. 4

3 Setup

The setup uses ImageNet-1K and ImageNet-100, Stable Diffusion generations, and classifiers spanning several architectures; Figure 2 compares real-only, generated-only, and mixed-data training.

3 Setup

  • ImageNet-1K provides 1.3 million training images and 50,000 validation images across 1,000 classes; ImageNet-100 supports many compute-limited analyses.The authors report similar trends on ImageNet-100 and ImageNet-1K.Source: §3 · p. 4
  • The classifier set includes ResNet-18, ResNeXt-50 and -101, EfficientNet-B0, and MobileNet-V2, with Stable Diffusion-V1-5 generating a 1.3M-image training set and 50K validation set.Generation is conditioned on proxy captions for class labels, using 80 diverse templates.Source: §3 · p. 4
  • Figure 2 compares ImageNet-1K validation and natural-shift accuracy for classifiers trained on real data, equal-sized generated data, or their full-data mixture; the mixture matches or outperforms either single source across various shifts.Across three random seeds, accuracy standard deviations ranged from 0.2 to 1 points.Source: Figure 2 · p. 5

4 Experiments

The experiments test generated data for training robust classifiers and evaluating classifiers on generated images. A real–generated mixture improves natural-shift performance, while results vary with augmentation strategy, dataset size, and conditioning.

4.1 Classification Accuracy and Robustness

  • A real–generated training mixture matches or outperforms real-only and generated-only training across natural-shift datasets, with ∼15% absolute gains on ImageNet-Sketch and ImageNet-R over standard training.It also preserves ImageNet-1K validation performance; generated-only training has high effective robustness but lower accuracy, while the mixture’s effective robustness exceeds standard training (= 0).Source: §4.1 · p. 6

4.2 Comparison with Standard Augmentations

  • Stable Diffusion generations yield the highest performance on ImageNet-R, ImageNet-Sketch, and ObjectNet among the compared strategies, while validation performance is similar across approaches.The comparison includes DeepAugment, PixMix, and a class-conditional latent diffusion model on overlapping ImageNet-100 classes.Source: §4.2 · p. 7, Table 3 · p. 7

4.3 Effect of Real and Generated Dataset Size

  • Increasing the amount of either real or generated training data improves accuracy on shifted datasets, and increasing generated data also raises effective robustness.The experiments vary real data from 25%, 50%, and 100% and generated data from 50%, 100%, and 200%.Source: §4.3 · pp. 7–8

4.4 Effectiveness of Generation Strategies

  • Diverse class-label templates and generation using both class labels and source images achieve closely tied performance at ∼66%, with no added benefit from source-domain information for zero-shot generation.Single-template generation has lower robustness than diverse templates, though it significantly outperforms real-only training in some domains.Source: §4.4 · p. 8

4.5 Evaluating Classifiers on Generated Datasets

  • ImageNet classifiers show substantial weaknesses on generated images: finetuned CLIP loses up to 17% and 12% absolute accuracy on original and filtered generated datasets, respectively.A classifier trained on real and generated data achieves up to 89% accuracy on real data and 97% on generated data.Source: §4.5 · pp. 8–9, Table 5 · p. 9

5 Generated Data from Finetuned Stable Diffusion

The section compares classifiers trained with generated data from base and ImageNet-finetuned Stable Diffusion. Despite reduced distribution gap, base-model generations yield higher accuracy gains on natural distribution shifts.

5 Generated Data from Finetuned Stable Diffusion

  • The comparison generates ImageNet-1K-scale data from finetuned Stable Diffusion and evaluates ResNext-50 classifiers on ImageNet-1K and natural-shift datasets.The setup includes classifiers trained only on finetuned generations and on an equal mix of real and finetuned-generated data.Source: §5 · p. 9
  • Classifiers trained solely on finetuned-generated data outperform those trained on base-generated data on ImageNet-1K validation, consistent with a reduced distribution gap between real data and generated data from finetuned Stable Diffusion.The finetuned Stable Diffusion classifiers achieve 38.1% accuracy, versus 67% reported for classifiers trained on data from finetuned ImaGen in [2]; the authors attribute the difference to the quality of the base generative models.Source: §5 · p. 9
  • 78% ImageNet-1K validation accuracy with real plus finetuned-generated data is close to 78.4% with real plus base-generated data.The authors state that the reason for this result is unclear.Source: §5 · p. 10
  • Base-generated data yields higher accuracy gains on natural distribution shifts than finetuned-generated data, including 40.1% on ImageNet-Sketch and 56.2% on ImageNet-R for the base-data classifier.The corresponding finetuned-data classifier reports 56.2% on ImageNet-Sketch and 41.5% on ImageNet-R.Source: §5 · p. 10

6 Related Work

Related work addresses robustness through engineered augmentations, multimodal pretraining, synthetic data, and evaluation on natural image variations. This paper uses generated data to train classifiers for natural distribution shifts and benchmarks existing classifiers on generated images.

6 Related Work

  • Prior augmentation research includes hand-engineered and learned policies, while DeepAugment and CLIP-based approaches have shown natural robustness; this work combines generative models with real datasets.The authors report that generated datasets outperform current augmentation strategies in eliciting robustness.Source: §6 · p. 10
  • Prior synthetic-data work examined adversarial robustness or CLIP accuracy, whereas this paper studies zero-shot generations for robust image classifiers under natural distribution shifts.The work also benchmarks existing image classifiers.Source: §6 · p. 10
  • Earlier evaluations document performance dips on natural image variations, while this work creates generated validation images and finds state-of-the-art ImageNet classifiers lose performance on them.The generated images depict new realizations of ImageNet-1K objects that may be difficult to acquire in the real world.Source: §6 · p. 10
  • Generated data has been used to diversify training examples and improve classification, while other work applies generation to commonsense corpora and image editing.This paper presents generated data as a practical means of improving classifier robustness.Source: §6 · p. 10

7 Conclusion

The paper presents a framework that augments real datasets with diverse generations from modern in-the-wild models and reports high performance on test and natural-shift datasets. It also uses synthetic data for evaluation, identifies brittleness in state-of-the-art models, and favors base-model generations over finetuned ones for robust training.

7 Conclusion

  • The framework augments real datasets with diverse data generated by modern in-the-wild models to improve image-classifier performance.The authors report high performance on test and natural distribution-shift datasets.Source: §7 · p. 10
  • Synthetic data also serves as an evaluation set, where state-of-the-art classifiers show brittleness to natural variations in generated images.The paper reports greater practical usefulness for base-model generations than finetuned generations for robust-classifier training.Source: §7 · p. 10
  • The authors identify a limitation in assessing generated-data trustworthiness through robustness alone and call for analysis of privacy and harmful stereotypes.They also note that the framework's computational cost includes dataset generation and classifier training, and that they lacked resources to study scaling laws.Source: §7 · p. 11

A Limitations

The limitations include unexamined fairness and privacy concerns and uncertainty about generalizing beyond ImageNet-1K. The paper also bounds its zero-shot approach to well-represented objects and discusses risks from web-trained generative models and generated data reuse.

A Limitations

  • The study focuses on ImageNet-1K objects, leaving generalization to larger datasets such as ImageNet-21K an open question.Fairness and privacy are not explored in this work.Source: A Limitations · p. 16
  • Zero-shot generation is focused on objects well represented in the base model's distribution; domains such as medical images may require fine-tuning.Source: A Limitations · p. 16

B Broader Impact

  • Web-scraped training data raises privacy and bias concerns, and generative models may amplify harmful biases during image generation.The authors call for closer curation of generated data and fairer multimodal representations.Source: B Broader Impact · p. 16
  • Generated data may be reused or scraped to train later models, making the sources of dataset biases difficult to track.The authors also note privacy concerns, including model generations replicating artists' styles.Source: B Broader Impact · p. 16

C Background - Data Generation using Stable Diffusion

The section describes Stable Diffusion as an in-the-wild generator and outlines ways to create synthetic images from class labels, source images, or both. The setup generated separate training and validation datasets for each strategy.

C Background - Data Generation using Stable Diffusion

  • Stable Diffusion is presented as an in-the-wild generator trained on diverse web-scraped image-caption pairs, capable of generating images from broad natural-language concepts and applying novel transformations.Its latent-diffusion pipeline encodes an image, denoises latent representations under conditioning, and decodes them into synthetic images.Source: C Background - Data Generation using… · p. 16
  • Class-label generation denoises random noise conditioned on CLIP text embeddings of natural-language templates for each class.The paper focuses primarily on this default Stable Diffusion strategy for studying natural robustness.Source: C Background - Data Generation using… · pp. 16–17
  • Source-image generation conditions denoising on CLIP vision-encoder representations to produce variations of source images.Source: C Background - Data Generation using… · p. 17
  • The combined strategy forward-diffuses an encoded source image, then denoises it conditioned on the class-label description.Figure 4 summarizes the three strategies: class labels, source images, and both together.Source: C Background - Data Generation using… · p. 17, Figure 4 · p. 17

D Setup

  • Each generation strategy used separate 130K-image training and 5K-image validation datasets; producing the complete dataset took about 10 days on five Nvidia RTX A5000 GPUs.Source: D Setup · p. 17

E Generated Data Analysis

The section evaluates generated-image properties and describes the evaluation and training setup. It also reports ImageNet-C results for classifiers trained with generated data.

E.1 Human Evaluation

  • The human study assessed whether generated images retained class-relevant information and image quality, using 1,600 annotations from 20 surveyors on 40 images across 10 ImageNet classes.Annotators saw images without source information, judged class-label consistency, and rated quality on a 1–5 scale.Source: E Generated Data Analysis · p. 17, §E.1 · p. 18, Figure 5 · p. 18

E Generated Data Analysis

  • Real ImageNet images score higher in consistency, quality, and diversity than class-label generations, while source-image generations are most diverse but score low in consistency and quality.Class-label generations score relatively close to real images, and class-label and combined source-image/class-label generations have similar consistency and quality scores.Source: Table 7 · p. 17, E Generated Data Analysis · p. 18

F Setup for Training Image Classifiers

  • Classifier training used FFCV dataloaders, batch size 512, and SGD; ImageNet-1K models trained for 40 epochs and ImageNet-100 models for 88.The learning rate was 0.5 with a cyclic schedule and weight decay of 5e-5.Source: F Setup for Training Image Classifiers · pp. 18–19

G More Details on Natural Distribution Shift Datasets

  • The evaluated natural shifts include sketches, renditions, a matched-frequency ImageNet-1K reproduction, and images with novel backgrounds and rotations.ImageNet-R includes 200 classes, 19 overlapping with ImageNet-100; ObjectNet has 113 overlapping ImageNet-1K classes and 13 overlapping ImageNet-100 classes.Source: G More Details on Natural Distribution… · p. 19

H Templates used for Data Generation

I Visualization of Image Generations

J ImageNet-C

  • On ImageNet-C at severity 5, classifiers trained on generated-only data or a real/generated mix achieve higher effective robustness than standard real-data training, while the mix raises ImageNet-1K validation accuracy by 1.25%.The table compares real, generated, and equal-mix training; the reported accuracy increase is on ImageNet-1K validation.Source: J ImageNet-C · p. 20, Table 9 · p. 20

K Effect of changing the training size

The section examines how varying real and generated data proportions relates to accuracy and effective robustness. In a fixed-budget ImageNet1K experiment, the two metrics change differently as the generated-data share increases.

K Effect of changing the training size

  • The ImageNet-100 training-size analysis varies the proportions of real and class-label-generated data and evaluates accuracy and effective robustness on ImageNet-Sketch, ImageNet-R, and ImageNet-V2.The figures define 100% as 130K training samples; effective robustness is measured relative to standard training on 100% real data.Source: Figure 7 · p. 20, K Effect of changing the training size · p. 20, Figure 8 · p. 21

K.1 Fixing the Amount of Training Data

  • With a fixed 1.3M-sample ImageNet1K budget, accuracy initially rises as generated data increases but drops 15% when its share goes from 0.75 to 1, while effective robustness rises monotonically.The experiment reports ResNeXt-50 results averaged over four natural distribution-shift datasets.Source: §K.1 · p. 21

L Training Dynamics

The section compares training dynamics for real ImageNet-1K data and an equal real/generated mixture. The accompanying figures describe data-proportion comparisons and loss curves.

L Training Dynamics

  • The ObjectNet figure varies the real ImageNet-100 and class-label-generated proportions, with 100% defined as 130K training samples.Its effective-robustness reference is standard training on 100% real data.Source: Figure 10 · p. 22
  • The ImageNet1K data-proportion figure fixes the training set at 1.3M samples and reports ResNeXt-50 results over three random seeds.It plots accuracy and effective robustness as the generated-data proportion varies.Source: Figure 11 · p. 22
  • The loss-curve comparison uses ResNeXt-50 trained on 1.3M real samples versus 2.6M samples in a 100:100 real/generated mixture.Source: L Training Dynamics · p. 21, Figure 12 · p. 22
Loading 2302.02503v2…