Source-linked AI summary
Diversify Your Vision Datasets with Automatic Diffusion-Based Augmentation
Lisa Dunlap, Alyssa Umino, Han Zhang, Jiezhi Yang, Joseph E. Gonzalez, Trevor Darrell
TL;DR
Specialized classifiers can struggle with domain changes, while collecting and labeling data from target domains can be costly. ALIA derives domain descriptions from training images, edits those images with language guidance, and filters poor edits. Its generated data outperforms traditional augmentation and text-to-image data by up to 7% across the evaluated tasks.
Problem
Specialized classification datasets can be limited in domain diversity, and collecting and labeling additional test-domain data is costly and difficult to target.
Source: §1 · p. 1Method
Results
ALIA-generated data outperforms traditional augmentation and text-to-image data by up to 7% across CUB, iWildCam, and Waterbirds, and beats adding real data on iWildCam.
Source: §1 · p. 2Takeaways & Limitations
The paper presents domain descriptions and augmented data grounded in the training set as an effective strategy for increasing dataset diversity in challenging settings.
Source: §7 · p. 10ALIA’s performance depends on its captioning, language, and editing models; text alone may not reliably change task-specific features such as a bird’s pose, and the optimal augmentation quantity remains unresolved.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Many fine-grained classification tasks, like rare animal identification, have limited training data and consequently classifiers trained on these datasets often fail to generalize to variations in the domain like changes in weather or location. As such, we explore how natural language descriptions of the domains seen in training data can be used with large vision models trained on diverse pretraining datasets to generate useful variations of the training data. We introduce ALIA (Automated Language-guided Image Augmentation), a method which utilizes large vision and language models to automatically generate natural language descriptions of a dataset's domains and augment the training data via language-guided image editing. To maintain data integrity, a model trained on the original dataset filters out minimal image edits and those which corrupt class-relevant information. The resulting dataset is visually consistent with the original training data and offers significantly enhanced diversity. We show that ALIA is able to surpasses traditional data augmentation and text-to-image generated data on fine-grained classification tasks, including cases of domain generalization and contextual bias. Code is available at https://github.com/lisadunlap/ALIA.
1 Introduction
Specialized classifiers can fail on domain changes, while collecting and labeling test-domain data is costly. ALIA uses training-set descriptions to edit images and filter failures, producing diverse, task-relevant data; its additions outperform traditional augmentation and text-to-image data by up to 7%.
1 Introduction
- The approach addresses costly test-domain data collection, which is difficult to target when the relevant domains are unknown.Specialized classifiers can fail under changes such as weather or location.Source: §1 · p. 1
- Unlike text-only generation, editing grounded in training images aims to vary domains while preserving task-relevant information and visual consistency.The overview contrasts ALIA with text-to-image and traditional augmentation examples on CUB.Source: §1 · pp. 1–2, Figure 1 · p. 2
- ALIA captions training images, summarizes their domains into prompts, edits the images, and filters edits that fail to change domains or corrupt task-relevant information.The approach requires neither model finetuning nor user-supplied prompts.Source: §1 · p. 2
- ALIA-generated data outperforms traditional augmentation and text-to-image data by up to 7% across fine-grained classification, domain generalization, and contextual-bias evaluations.The evaluations use CUB, iWildCam, and Waterbirds; ALIA also beats adding real data on iWildCam.Source: §1 · p. 2
2 Related Works
Prior augmentation work uses generative models, conventional image transformations, or language-guided approaches. ALIA distinguishes itself by grounding both domain descriptions and image edits in the provided training set.
Supplementing Training Data with Generative Models
- Generative-model augmentation spans medicine [8] [33], domain adaptation, and bias mitigation [34], but diffusion-generated training data can underperform real data without task-specific finetuning.ALIA instead uses diffusion for text-guided image editing to keep augmentations close to training data without finetuning.Source: §2 · p. 2
Traditional Data Augmentation
Language-guided Data Augmentation for Classification
3 ALIA: Automated Language-guided Image Editing
ALIA augments labeled training data with edits that represent image domains while retaining task information. It derives domain prompts from captions, edits original images with text guidance, and filters poor edits before adding selected augmentations back to training.
3 ALIA: Automated Language-guided Image Editing
- ALIA aims to represent image domains—such as location, weather, or time of day—through edits that improve their coverage in a labeled dataset.The method defines a domain as an image aspect not intended for classification.Source: §3 · p. 3
3.1 Generating Domain Descriptions
- The method captions dataset images and summarizes the captions into fewer than 10 domain descriptions, often including settings seen in test data.Captions can describe context rather than task-specific information; the authors sample 200 unique captions for summarization.Source: §3.1 · p. 4
3.2 Editing Images with Language Guidance
- Stable Diffusion-based Img2Img edits original images conditioned on text, while Instruct Pix2Pix generates edits from an image and an edit instruction.ALIA can also use other text-conditioned image-editing methods.Source: §3.2 · p. 4
3.3 Filtering Failed Edits Using Semantic and Visual Features
- Filtering targets total failures, nearly unchanged edits, and edits that alter class-relevant features; confidence filtering uses a classifier trained on the original data to detect the latter two.CLIP semantic filtering removes images unrelated to the task, while confidence thresholds remove edits confidently predicted as unchanged or as another class.Source: §3.3 · pp. 4–5
- The authors sample filtered edits to expand training data by 20–100%, aiming to preserve visual consistency while covering more domains.The prompts and image edits are grounded in the original training set.Source: §3.3 · p. 5
4 Experimental setup
The experiments fine-tune ResNet50 models on CUB, iWildCam, and Waterbirds and compare ALIA additions against real data, text-to-image data, and established augmentation baselines. Model and editing parameters are selected using validation performance and visual inspection.
4.1 Implementation.
- Experiments fine-tune ImageNet-pretrained ResNet50 models for CUB, iWildCam, and Waterbirds, selecting learning rate and weight decay by validation performance.Training uses Adam, a cosine learning-rate scheduler, and 10 GeForce RTX 2080 Ti GPUs.Source: §4.1 · p. 5
- Diffusion editing parameters are chosen by inspecting generated images for diversity, retained task information, and faithfulness to instructions; reported results average three random seeds.The search tests five values each for edit strength and text guidance across 40 generated images.Source: §4.1 · p. 5
4.2 Baselines.
- Baselines include the original training set, added real test-distribution data, text-to-image data, CutMix, and RandAugment, with added image counts matched across methods.ALIA and text-to-image samples are generated in excess before filtering, then sampled to match the real-data class distribution.Source: §4.2 · p. 6
5 Experiments
The experiments evaluate ALIA on iWildCam domain generalization, CUB fine-grained bird classification, and Waterbirds contextual bias. They also examine prompt and filtering choices, editing methods, and the amount of generated data added.
5.1 Domain Generalization [iWildCam [17]]
- 17% performance improvement over original training data on iWildCam, with ALIA also exceeding the same-amount real-data baseline.The authors note that the real-data comparison may partly reflect prompts drawn from empty images in the test domain.Source: Figure 4 · p. 6, §5.1 · p. 7
5.2 Fine-grained Classification [CUB [41]]
- On CUB, ALIA outperforms all baselines except RandAug and added real data; text-to-image data performs similarly to the baseline.The results also show improvements in conditions without domain shifts.Source: Figure 5 · p. 7, §5.2 · p. 7
5.3 Contextual Bias [Waterbirds [32]]
- 7% improvement in class-balanced accuracy over other augmentation baselines on Waterbirds, with roughly matched in-domain accuracy and improved out-of-domain accuracy.For overall accuracy, ALIA outperforms the other methods except the real-data baseline.Source: Figure 6 · p. 8, §5.3 · p. 8
5.4 Ablations
- ALIA-generated descriptions outperform user-provided prompts, and semantic and confidence-based filtering further improves accuracy.Table 1 evaluates generated prompts and filtering.Source: §5.4 · p. 8, Table 1 · p. 8
- 76.23±4.78% for InstructPix2Pix edits versus 84.87 ± 1.92% for Img2Img edits on iWildCam.InstructPix2Pix produced greater color variation but often added an artificial airbrush effect or removed original image detail.Source: §5.4 · p. 9, Figure 7 · p. 9
- On CUB, ALIA gains accuracy up to 1,000 generated images, then declines; text-only data decreases accuracy almost immediately and falls below baseline at 2,000 images.One thousand images are 20% of the original training set.Source: Figure 8 · p. 9, §5.4 · p. 9
6 Limitations
ALIA’s effectiveness depends on the quality of its captioning, language, and image-editing models. Text alone is unlikely to support edits to task-specific features such as bird pose, and the optimal amount of augmented data remains unresolved.
6 Limitations
- ALIA’s performance is bottlenecked by the quality of its captioning model, LLM, and image-editing method.The authors refer to their Section 5.4 investigation of filtering and editing choices.Source: §6 · p. 9
- Because task-specific information is assumed difficult to generate from text alone, changing a bird’s pose is likely to result in a failed edit.Source: §6 · p. 9
- The optimal quantity of augmented data to add back into the training set remains unresolved.Source: §6 · p. 9
7 Conclusion
The paper presents ALIA, which combines large language models’ domain knowledge with text-conditioned image editing and grounds descriptions and augmented data in the training set. It reports capabilities across domain adaptation, bias mitigation, and settings without domain shift, and identifies structured prompts from training data as a potential strategy for improving dataset diversity.
7 Conclusion
- ALIA combines large language models’ high-level domain knowledge with text-conditioned image editing for data augmentation.Source: §7 · p. 10
- Grounding domain descriptions and augmented training data in the provided training set is central to the approach.Source: §7 · p. 10
- The paper reports capabilities in domain adaptation, bias mitigation, and scenarios without domain shift.Source: §7 · p. 10
- The authors suggest structured prompts generated from actual training data as a potential strategy for improving dataset diversity.They expect the approach’s efficacy and scope to increase as captioning, LLM, and image-editing methods improve.Source: §7 · p. 10
8 Dataset Details
The appendix documents dataset split counts, generation-time procedures, and construction of the iWildCam subset. It explains the subset’s constraints and how its non-overlapping splits were selected before experimentation.
8.1 Image Counts
- Table 2 reports dataset split sizes, with CUB’s validation and test sets identical; Table 3 provides Waterbirds statistics, and extra real-baseline data were constructed manually.Source: §8.1 · p. 13, Table 2 · p. 13, Table 3 · p. 13
8.2 Time to Generate Image Edits
- ALIA generates edits at twice the dataset size to leave enough filtered images to match the real-data baseline; a pilot edit-and-filter run can estimate the needed volume.Table 4 presents generation times for two edits per dataset, but the supplied passage gives no time values.Source: §8.2 · p. 13, Table 4 · p. 13
8.3 iWildCam Subset Construction
- The iWildCam subset limits experiment cost and ensures sufficient class examples, while keeping train, validation, test, and extra splits in non-overlapping locations.Classes were not artificially balanced; sampled locations came from train and id_test for train and validation, while disjoint groups of test locations formed test and extra splits before experimentation.Source: §8.3 · p. 13
- Figure 9 shows background-image samples from the two locations used for the iWildCam subset test set.Source: §8.3 · p. 13, Figure 9 · p. 14
9 Hyperparameters
Hyperparameters were selected using baseline validation accuracy and then held constant across methods.
9 Hyperparameters
- The baseline’s highest-validation-accuracy configuration was selected from five learning rates and four weight decays, then used across all methods.Table 6 lists the selected hyperparameters; the passage does not provide their values.Source: §9 · p. 14, Table 6 · p. 14
10 Additional Visualizations
ALIA’s filtering removes failed edits, while stronger image editing and text guidance produce more pronounced context changes but can corrupt class information. In the reported comparison, the stronger settings achieved higher accuracy; unfiltered samples also show poor-quality Waterbirds edits.
10.1 Filtered Images
- The semantic and confidence filters remove failed edits, including irregular images, edits too similar to the original, and edits that corrupt task-relevant information.The examples include high-confidence edits with both correct and incorrect predictions.Source: §10.1 · pp. 14–15
10.2 Different Strengths and Text Guidance
- Higher edit strength moves outputs farther from originals, while higher text guidance makes prompted content more evident; both may corrupt class information.Figure 11 varies edit strength, and Figure 12 varies text guidance.Source: §10.2 · p. 15, Figure 11 · p. 15, Figure 12 · p. 16
- 84.87% accuracy at strength 0.6 and text guidance 7.5, versus 76.11% at strength 0.3 and text guidance 5.0.The authors report an approximate 8-point drop with the lower settings.Source: §10.2 · p. 15
10.3 Generated Images
- Unfiltered iWildCam and CUB examples show edits across prompts, while Waterbirds examples include edits in which the bird is heavily corrupted or removed.The iWildCam and CUB samples precede filtering; the Waterbirds examples show poor-quality Img2Img edits.Source: §10.3 · p. 16, Figure 13 · p. 17, Figure 14 · p. 18