Effective Data Augmentation With Diffusion Models

Brandon Trabucco; Kyle Doherty; Max Gurinas; Ruslan Salakhutdinov

Source-linked AI summary

Effective Data Augmentation With Diffusion Models

Brandon Trabucco, Kyle Doherty, Max Gurinas, Ruslan Salakhutdinov

arXiv:2302.07944v3Feb 7, 2023cs.CVcs.AI

TL;DR

Standard augmentations do not vary key semantic attributes, limiting diversity from few real images. DA-Fusion edits real images with a pretrained text-to-image diffusion model, and improves few-shot classification accuracy by up to +10 percentage points in tested domains. The method also produces gains on a weed-recognition dataset outside the diffusion model’s vocabulary, while its augmentation control remains limited.

Problem
Standard flips and rotations do not alter high-level semantic attributes or create novel structural elements, textures, and perspectives.
Method
DA-Fusion adapts a pretrained diffusion model with pseudo-prompts to semantically modify real images while balancing real and synthetic data.
Results
+10 percentage points is the maximum reported improvement in few-shot classification accuracy across tested domains, with gains also reported for weed recognition outside the model’s vocabulary.
Takeaways & Limitations
DA-Fusion improves few-shot classification in the tested domains, including common, fine-grain, novel, and previously unseen weed-recognition concepts.
Takeaways & Limitations
The method does not explicitly control how the diffusion model augments an image, such as changing a cat’s breed.

Abstract

from arXiv · show

Data augmentation is one of the most prevalent tools in deep learning, underpinning many recent advances, including those from classification, generative models, and representation learning. The standard approach to data augmentation combines simple transformations like rotations and flips to generate new images from existing ones. However, these new images lack diversity along key semantic axes present in the data. Current augmentations cannot alter the high-level semantic attributes, such as animal species present in a scene, to enhance the diversity of data. We address the lack of diversity in data augmentation with image-to-image transformations parameterized by pre-trained text-to-image diffusion models. Our method edits images to change their semantics using an off-the-shelf diffusion model, and generalizes to novel visual concepts from a few labelled examples. We evaluate our approach on few-shot image classification tasks, and on a real-world weed recognition task, and observe an improvement in accuracy in tested domains.

1 INTRODUCTION

Standard augmentations improve robustness to color and geometry but do not create novel semantic structure. DA-Fusion uses pretrained text-to-image diffusion models to generate semantically varied augmentations, improving few-shot classification by up to +10 percentage points.

Standard transformations such as flips and rotations improve robustness to color and geometry but do not add novel structural elements, textures, or perspectives.
DA-Fusion generates variations of real images with text-to-image diffusion models, complementing standard augmentation while retaining universality, controllability, and performance.
The method adapts to new domains by fine-tuning pseudo-prompts in the text encoder for the concepts being augmented.
DA-Fusion modifies object appearance while respecting semantic invariances, including changes to graffiti or train design.
+10 percentage points is the maximum reported improvement from DA-Fusion across tested few-shot classification domains, including a real-world weed-recognition task.

2 RELATED WORK

Prior work uses generative models for synthetic data and image editing, but this paper focuses on diffusion-based augmentation for previously unseen concepts. It presents this as the first analysis of synthetic data on such concepts.

Image-editing methods include masked inpainting, attention-weight modification, and SDEdit, which inserts real images partway through reverse diffusion.
Generative models have been used for representation learning, inverse graphics, semantic segmentation, and classifier training with synthetic data.
Diffusion-based synthetic data has recently been studied in few-shot settings, where models likely saw images of target classes during training.
The paper identifies its contribution as the first analysis of synthetic data for previously unseen concepts.

3 BACKGROUND

Diffusion models generate samples by reversing a learned Gaussian Markov process from noise to data. Stable Diffusion adds a text encoder for text-to-image generation.

Diffusion models are sequential latent-variable models that generate samples through a Markov chain of learned Gaussian transitions from initial noise.
Reverse transitions progressively reduce variance according to β1, . . . , βT so the final sample x0 represents a sample from the true distribution.
The transitions use fixed covariance Σt = βtI and a learned mean μθ(xt, t).
The neural network ϵθ processes a noisy sample xt and predicts the added noise, while real samples x0 and noise ϵ generate xt at arbitrary timesteps.
The method uses a pretrained Stable Diffusion model whose text encoder enables text-to-image generation.

4 DATA AUGMENTATION WITH DIFFUSION MODELS

DA-Fusion uses text-to-image diffusion models to generate controllable, semantically varied augmentations from real images, while mixing synthetic and real data for training.

4 DATA AUGMENTATION WITH DIFFUSION MODELS: The method is designed to be universal, controllable, and performant across new and fine-grained concepts.These desiderata include high-fidelity augmentation, simple control of content and extent, and accuracy gains that justify generation cost.
4.1 A UNIVERSAL GENERATIVE DATA AUGMENTATION: DA-Fusion adapts diffusion models to new concepts by inserting and fine-tuning pseudo-prompt embeddings in the text encoder.This allows augmentation of concepts outside the pretrained model’s vocabulary.
4.1 A UNIVERSAL GENERATIVE DATA AUGMENTATION: Real images guide generation by being spliced into reverse diffusion at timestep ⌊St0⌋, after which reverse diffusion produces edited samples.The insertion position t0 controls how much the source image is transformed.
4.2 CONTROLLING AUGMENTATION: Synthetic and real images are balanced during training by sampling a synthetic image with probability α, with α = 0.5 effective across tested domains.This prevents training from relying exclusively on synthetic data.
4.1 A UNIVERSAL GENERATIVE DATA AUGMENTATION: Each real image produces M augmentations, forming a synthetic dataset whose samples are indexed alongside the N real images.Indices are sampled uniformly across real images and their augmentations.
4.2 CONTROLLING AUGMENTATION: Randomly sampling the insertion timestep increases augmentation diversity, with k = 4 consistently improving classification accuracy over a fixed t0.The authors identify guidance-scale randomization as a possible alternative for future work.

5 DATA PREPARATION

The evaluation uses six standard computer-vision datasets and a newly contributed leafy-spurge drone-image dataset whose aerial domain is distinct from retrieved diffusion-model imagery.

Standard Datasets: The benchmark covers Caltech101, Flowers102, FGVC Aircraft, Stanford Cars, COCO, and PASCAL VOC.COCO uses official 2017 training and validation sets, while PASCAL VOC uses official 2012 sets.
Leafy Spurge: The contributed Leafy Spurge dataset contains top-down drone images of semi-natural areas in the western United States.It targets Euphorbia esula, an invasive plant affecting natural and agricultural ecosystems.
Leafy Spurge: CLIP retrieval found close-up images from the same genus among the top 20 results and the target species at rank 35, but no top-down aerial examples.The retrieved imagery therefore does not match the dataset’s aerial viewpoint and landscape context.

6 DA-FUSION IMPROVES FEW-SHOT CLASSIFICATION

DA-Fusion improves few-shot classification across common, fine-grain, and novel concepts, while addressing leakage concerns through model- and data-centric defenses. Its gains remain robust across real–synthetic data balances, with M = 10 offering a practical trade-off.

Few-shot classification: DA-Fusion improves accuracy by +5 to +15 points across all seven evaluated domains compared with standard data augmentation.The evaluation includes common, fine-grain, and completely new concepts.
Concept novelty: DA-Fusion uses optimized pseudo-prompts to guide diffusion augmentation without requiring prompt engineering for difficult concepts.The pseudo-prompt format is “a photo of a <wi>,” with the embedding optimized for the target concept.
Concept novelty: 12.8%, 24.2%, and 20.8% improvements are reported for common, fine-grain, and novel concepts, respectively.Real Guidance is described as failing systematically on novel and fine-grain concepts, whereas DA-Fusion works across novelty levels.
Preventing leakage of internet data: Leakage-prevention experiments show gains of up to +5 points with model-centric erasure and up to +10 points with data-centric input masking on Pascal and COCO.DA-Fusion exceeds Real Guidance under both defenses while using the same hyperparameters.
Preventing leakage of internet data: The larger gains under the weaker data-centric defense suggest that part of DA-Fusion’s benefit comes from Stable Diffusion’s prior class knowledge, motivating strong leakage prevention.The data-centric defense removes class mentions from model inputs, while the model-centric defense edits model weights.
DA-Fusion is robust to data balance: Performance is stable across α and M settings; α = 0.7 and M = 20 are marginally better, but M = 10 avoids doubling generative-model calls for marginal improvement.The paper uses α = 0.5 and M = 10 throughout and presents this as sufficient for practical tuning.

7 DISCUSSION

DA-Fusion consistently improves few-shot classification accuracy under both leakage defenses and across tested domains. The method remains promising, but greater control over edits, temporal consistency, and diffusion-model fidelity are identified as future directions.

DA-Fusion consistently improves few-shot classification accuracy when evaluated with both model-centric and data-centric leakage defenses.
The method improves accuracy by up to +10 percentage points across various datasets and also gains on a weed-recognition dataset outside the diffusion model’s vocabulary.
Future work should improve control over object modifications, including independently and more extremely changing selected attributes.
Maintaining temporal consistency is an important challenge for applying DA-Fusion in decision-making settings such as visual reinforcement learning.
Improving the diffusion-model backbone’s photo-realism is expected to improve DA-Fusion.

A LIMITATIONS & SAFEGUARDS

The paper addresses risks from harmful or copied generative content through safety checking, real-image editing, and concept deletion. These safeguards reduce risk but do not fully resolve bias, copyright, privacy, or bodily-autonomy concerns.

Safeguards: A safety checker discards and resamples augmented images containing nudity or violence.
Safeguards: Editing real images preserves their high-level structure and lowers the risk of introducing harmful content when the source images are clean.
Safeguards: The combined safeguards lower, but do not eliminate, the risk of harmful or biased synthetic content entering augmented data.
Ethical considerations: Deleting harmful or copyrighted concepts from Stable Diffusion weights is proposed to reduce the chance that augmentations copy users’ online data.
Broader impacts: Few-shot augmentation can reduce the data needed for systems that target particular people, creating privacy and personal-safety risks.
Additional safeguards evaluation: DA-Fusion improves over RandAugment and CutMix under their respective leakage-prevention regimes on the Pascal-based task.
Additional safeguards evaluation: On additional datasets, DA-Fusion outperforms standard augmentation and a competitive recent method under both leakage-prevention approaches.

D ADDITIONAL RESULTS

Additional experiments show that DA-Fusion improves few-shot classification on standard Caltech101 and Flowers102 benchmarks and surpasses stronger augmentation baselines on the Pascal-based task.

DA-Fusion improves classification performance on the Caltech101 and Flowers102 few-shot tasks under both model-centric and data-centric leakage defenses.
DA-Fusion improves over both RandAugment and CutMix on the Pascal-based task.

F DIFFERENT CLASSIFIER ARCHITECTURES

DA-Fusion’s gains extend beyond the ResNet50 classifier to the DeiT architecture on the Pascal task. The evaluation uses data-centric leakage prevention and the method’s specified hyperparameter configuration.

DA-Fusion improves DeiT performance over the standard data-augmentation baseline and Real Guidance on the Pascal task.
The DeiT result suggests that DA-Fusion’s gains generalize across convolution-based and attention-based classifier architectures.
Figure 14 reports the stronger-classifier comparison for DeiT with data-centric leakage prevention on Pascal.
The method inherits hyperparameters from text-to-image diffusion models and SDEdit, while additional hyperparameters control synthetic-image diversity.
The reported evaluation uniformly selects 20 classes per dataset, producing 20-way classification tasks to reduce reproduction cost.

H LEAFY SPURGE DATASET ACQUISITION AND PRE-PROCESSING

The leafy spurge dataset combines botanist-verified drone surveys with plot-level cropping and synthetic-image examples generated using DA-Fusion methods.

Dataset acquisition: 39 surveyed sites in western Montana were selected across elevation and solar-input conditions to represent environmental variation.Botanists verified leafy spurge presence or absence at each site.
Dataset acquisition: Each site was divided into a 3 x 3 grid of 10m plots, with leafy spurge presence or absence recorded for every cell.Drone RGB images were collected at 50m above each site under sunny afternoon lighting.
Pre-processing: Plot images were cropped using survey markers, producing 533–1059-pixel sides with a mean side length of 717 pixels and 1.4 cm mean ground sampling distance.Terrain caused crop sizes to vary because ridges were closer to the drone than valleys.
Pre-processing: Four 250x250-pixel crops were generated around each plot center because smaller crops classified better than plot-scale images.A botanist filtered positive crops where cropping excluded or obscured target plants.
Synthetic augmentation: DA-Fusion examples were generated using pooled class-level tokens or image-specific tokens, with source images and inference settings held otherwise identical within columns.The two token-generation strategies are illustrated on separate rows.

I BENCHMARKING THE LEAFY SPURGE DATASET

The benchmark compares legacy flips and rotations with pooled and image-specific DA-Fusion augmentation on the full leafy spurge dataset. Both DA-Fusion variants improve accuracy, while the image-specific approach gains slightly more at higher computational cost.

Benchmark design: 15 random 20%-validation trials fine-tuned ResNet50 for 500 epochs and reported cross-validated mean accuracy with 68% confidence intervals.Each trial trained on the remaining 80% of the dataset.
Benchmark design: Baseline augmentation used horizontal and vertical flips plus random rotations up to 45 degrees with probability 0.5.These legacy transformations formed the comparison condition.
Results: 1.0% gain came from DA-Fusion Pooled and 1.2% gain from DA-Fusion Specific over baseline on the full leafy spurge dataset.Both methods produced slight performance enhancements, with the specific approach performing better in this experiment.
Interpretation: Pooling may be less beneficial because negative examples represent diverse backgrounds rather than one semantic concept.The authors suggest a hybrid approach using pooled positive tokens and specific negative tokens for further investigation.