PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Objects: Contextual Synthetic Data Generation for Fine-Grained Classification

William Yang, Xindi Wu, Zhiwei Deng, Esin Tureci, Olga Russakovsky

arXiv:2510.24078v2cs.CV

TL;DR

Fine-grained classification needs synthetic images that preserve subtle class distinctions, but few-shot T2I fine-tuning risks overfitting and reduced diversity. BOB conditions fine-tuning on extracted background and pose, then marginalizes those attributes during generation. It outperforms prior art by at least 2% in 18 of 24 experimental settings; the paper also flags data-representation and real-world-use concerns.

  • Problem

    Fine-grained classification depends on subtle visual differences, while T2I distribution misalignment can produce artifacts or incorrect compositions that hinder recognition.

    Source: §1 · p. 1

  • Method

    BOB extracts class-agnostic background and pose, conditions T2I fine-tuning on these attributes, and samples them across classes during generation.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    BOB outperforms prior art by at least 2% in 18 of 24 experimental settings.

    Source: §1 · p. 2

  • Takeaways & Limitations

    In three of four reported datasets, five real images augmented with BOB-generated images outperform using ten real images.

    Source: §1 · p. 2

    Few-shot data may carry representation biases, and downstream effects in real-world applications require detailed study.

    Source: Ethics Statement · p. 9

Abstract

from arXiv · show

Text-to-image (T2I) models are increasingly used for synthetic dataset generation, but generating effective synthetic training data for classification remains challenging. Fine-tuning a T2I model with a few real examples can help improve the quality of synthetic training data; however, it may also cause overfitting and reduce diversity in the generated samples. We propose a fine-tuning strategy BOB (BeyondOBjects) to mitigate these concerns for fine-grained classification. Given a small set of real examples, we first extract class-agnostic attributes such as scene background and object pose. We then explicitly condition on these attributes during fine-tuning of the T2I model and marginalize them out during generation. This design mitigates overfitting, preserves the T2I model's generative prior, reduces estimation errors, and further minimizes unintended inter-class associations. Extensive experiments across multiple T2I models, backbones, and datasets show that our method achieves state-of-the-art performance in low-shot fine-grained classification when augmented with synthetic data. Concretely, BOB outperforms DataDream by 7.4% on the Aircraft dataset (from 50.0% to 57.4% when fine-tuning a CLIP classifier with five real images augmented with 100 synthetic images). In three of the four benchmarks, fine-tuning downstream models with 5 real images augmented with BOB achieves better performance than fine-tuning with 10 real images. Collectively, BOB outperforms prior art in 18 of 24 experimental settings, with 2+% accuracy improvements in 14 of these settings.

1. Introduction

Fine-grained classification is difficult to support with synthetic T2I data because subtle class differences and few-shot fine-tuning create challenges for accurate, diverse generation. BOB conditions fine-tuning on class-agnostic background and pose attributes and samples those attributes across classes, with strong results across the reported settings.

1. Introduction

  • Fine-grained classification requires distinguishing subtle visual differences, while T2I estimation errors can produce artifacts or incorrect compositions that hinder recognition.The paper illustrates this with the subtle winglet and wing-length difference between two Boeing 747 variants.Source: §1 · p. 1
  • Few-shot fine-tuning can improve generated-image accuracy and diversity, but may overfit, erode the T2I model’s world prior, and reduce synthetic-data diversity.The figure contrasts prior methods that omit rich context in fine-tuning and sampling with BOB’s use of context in both stages.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • BOB extracts class-agnostic background and pose attributes, uses them in T2I fine-tuning, and samples them across the dataset during generation.The sampling is intended to marginalize unintended inter-class associations.Source: §1 · p. 2
  • BOB outperforms prior methods by at least 2% in 18 of 24 experimental settings, with comparable performance in the remaining six.For CLIP on Aircraft, five real images plus 100 BOB images yield 57.4% accuracy versus 50.0% with DataDream.Source: §1 · p. 2

2. Related Work

Prior work uses personalization and T2I generation for classification, but concept fidelity and limited variation can conflict with classifier training needs. BOB retains controllability during fine-tuning and marginalizes context during generation.

2. Related Work

  • Personalization methods guide T2I models with a few image exemplars, but their objectives differ from classification-oriented synthetic-data generation.The paper names several personalization approaches and notes their concept-level control.Source: §2 · p. 2
  • Personalization can reproduce specific visual concepts, but its emphasis on concept fidelity may limit intra-class variation and inter-class separability needed for classifier training.Pretrained T2I models are described as offering a world prior that can improve sample diversity and fidelity over interpolation-based augmentation.Source: §2 · p. 2
  • Prior synthetic-classification work improves either fine-tuning or generation, whereas BOB maintains T2I controllability during fine-tuning and shifts generation toward marginalizing spurious correlations.The overview describes extracting background and pose, conditioning fine-tuning on enriched captions, then sampling background-pose pairs across the dataset.Source: §2 · p. 2, Figure 2 · p. 3

3. Methodology

BOB addresses overfitting in both text and image generation through two stages: Context Preservation during fine-tuning and Context Marginalization during generation. The method enriches captions with background and pose, then samples those attributes independently of class labels.

3. Methodology

  • BOB uses Context Preservation during fine-tuning and Context Marginalization during generation to address text- and image-side overfitting in few-shot synthetic-data generation.The text-side challenge is that generic class labels fail to express intra-class visual range; the image-side challenge is unintended inter-class associations.Source: §3 · p. 3

3.1. Context Preservation

  • Context Preservation associates each image with a descriptive caption containing class-agnostic background and pose attributes, restoring visual context absent from generic class labels.The captioning model extracts background and pose using prompts that refer to a dataset-level descriptor rather than class-specific object details.Source: §3.1 · p. 3
  • The extracted background and pose are stored in a caption bank, and LoRA fine-tunes attention layers of both the U-Net and CLIP text encoder with the standard diffusion objective.The enriched image-text pairings condition the fine-tuned model on class and context attributes.Source: §3.1 · p. 3

3.2. Context Marginalization

  • Context Marginalization samples background-pose pairs independently of class labels, exposing each class to dataset-wide contextual variety.The method uses back-door adjustment to model P(X|do(Y)) by averaging over class-agnostic attributes Z; the causal graph frames images X as generated from class-relevant Y and class-agnostic Z.Source: §3.2 · p. 4, Figure 3 · p. 4

4. Experiments

Across few-shot and long-tail settings, BOB generally outperforms prior synthetic-data methods, with gains especially clear on Aircraft and imbalanced CUB-LT. Analyses associate its gains with better alignment to real data and the combination of context preservation and marginalization.

4.1. Experimental Setup

  • Evaluations cover four few-shot datasets, two long-tail datasets, and three downstream backbones, with 5 or 10 real images per class in the few-shot setting.The study compares seven existing methods and evaluates BOB with Stable Diffusion v1.5 and v2.1-base.Source: §4.1 · p. 4, §4.2 · p. 4

4.2. Few-shot Classification

  • BOB improves over the best prior method on Aircraft, Car, and CUB, with reported gains of 2.77%–10.25%; Pets is the exception.Across three downstream backbones, BOB obtains similar performance to prior methods on Pets.Source: §4.2 · p. 5, Table 1 · p. 5
  • On Aircraft with ResNet-50, BOB raises 5-shot accuracy from 39.62% to 60.31%, a 20.69% improvement and 5.73% more than DataDream’s improvement.With CLIP and MAE, BOB improves over the best prior method by 3.78%–7.33% in 5-shot and 4.65%–5.79% in 10-shot settings.Source: §4.2 · p. 5
  • With BOB augmentation, five real images outperform ten real images on Aircraft, Car, and CUB across three backbones; Pets is the exception, with similar performance.Reported examples include 60.31% versus 55.48% for Aircraft with ImageNet and 88.64% versus 78.50% for Car with ImageNet.Source: §4.2 · p. 6

4.3. Long-tail Classification

  • On CUB-LT, BOB beats prior methods by at least 4% across imbalance factors; on Flower-LT, it gains 2%–3% at IF=100 and is within 1% at IF=50 and IF=10.For CUB-LT at IF=100, BOB reaches 62.19% versus Diff-II’s 56.10%, and 63.06% versus DataDream’s 53.42%.Source: §4.3 · pp. 6–7, Table 2 · p. 6

4.4. Analysis

  • BOB’s synthetic data is closer to the training distribution: mean per-class FID has a mode around 26, versus 31 for DataDream and 37 for Diff-II.BOB lowers FID versus DataDream in 91 of 100 classes; the figure also contrasts this alignment with caption-model accuracy and downstream performance.Source: §4.4 · pp. 7–8, Figure 5 · p. 7
  • Caption-model fine-grained classification accuracy has no discernible association with downstream performance, and the authors conclude distillation does not explain BOB’s gains.GPT-4o’s captioning performance is close to 80%, but using it does not improve downstream classification performance.Source: Figure 5 · p. 7, §4.4 · p. 8
  • Prompt diversity matters but does not fully explain BOB’s gains; the ablations report the best accuracy when both context preservation and marginalization are used.GPT-generated contexts reach 72.10%; the combined preservation-and-marginalization setting reaches 73.78%, while preservation without marginalization drops to 65.90% from 68%.Source: §4.4 · p. 8, Table 3 · p. 8, Table 4 · p. 8

5. Conclusion

BOB conditions T2I fine-tuning on class-agnostic background and pose information, then marginalizes these attributes during generation. Experiments report gains across multiple backbones, datasets, and scales, while broader coarse-grained use and automatic attribute discovery remain challenging.

5. Conclusion

  • BOB reports consistent performance gains across multiple backbones, datasets, and scales, including over 7% improvement on Aircraft and best performance in nearly all settings.The conclusion describes caption-guided fine-tuning as having potential for downstream classification and identifies scaling to broader domains as further research.Source: §5 · p. 8
  • BOB extracts class-agnostic background and pose information, conditions on these attributes during fine-tuning, and marginalizes them during data generation.The conclusion says this design mitigates overfitting, reduces unintended class associations, and narrows the synthetic-to-real distribution gap.Source: §5 · p. 8
  • Automatic discovery of reliable class-agnostic attributes or invariances remains an open problem for BOB.Source: §5 · p. 8
  • Scaling BOB to coarse-grained recognition such as ImageNet remains challenging because classes become context-dependent and spurious correlations can inflate performance.Source: §5 · p. 8

Ethics Statement

The paper uses public fine-grained datasets for non-commercial research and cautions that limited geographic or environmental coverage may bias their representation. It also flags risks from few-shot data and misuse, recommending further study and watermarking.

Ethics Statement

  • The public datasets are used for non-commercial research, and their limited geographic or environmental context may create representation biases.The models trained on them are not intended for direct use without considering those representation issues.Source: Ethics Statement · p. 9
  • Few-shot data may naturally include biases, so downstream effects should be studied in detail for real-world applications.Source: Ethics Statement · p. 9
  • Because synthetic-data methods could be misused for privacy violations, copyright violations, or deepfakes, the authors encourage watermarking real and generated images.Source: Ethics Statement · p. 9

Classification

Table 5 documents fine-tuning hyperparameters for the T2I model and downstream classifier, including validation sweeps and the test-training epoch convention.

Classification

  • Table 5 covers hyperparameters for fine-tuning both the T2I model and the downstream classifier.Source: Table 5 · p. 13
  • Listed parameter values indicate hyperparameter sweeps selected using the validation set.Source: Table 5 · p. 13
  • The epoch count in parentheses indicates the test-set fine-tuning epochs using the best validation-selected hyperparameter.Source: Table 5 · p. 13

6. Additional implementation details

The implementation details describe fine-tuning and generation settings, along with validation-set construction for few-shot and long-tail evaluation.

6.1. Hyperparameter sweep

  • For T2I fine-tuning, the procedure follows DataDream except that it uses dense captions and increases training from 200 to 400 epochs.The authors select settings through validation sweeps for downstream classifiers, including 16 configurations for CLIP, 18 for ImageNet, and 8 for MAE.Source: §6.1 · p. 13
  • The selected configuration is first trained for 10 epochs, then the original pretrained model is retrained for 100 epochs using the best validation configuration.Test accuracy is reported at the epoch corresponding to the best validation accuracy.Source: §6.1 · p. 13

6.2. Parameters for dataset generation

  • Synthetic-data generation follows DataDream settings of guidance scale 2.0, 50 inference steps, and fp16 mixed precision, with the default scheduler for Stable Diffusion v1.5 and v2.1-base.Baselines use their papers' default generation settings; Diff-Mix additionally applies CLIP filtering.Source: §6.2 · p. 13, Table 6 · p. 13

6.3. Creation of validation dataset

  • Few-shot validation uses 16 non-overlapping images per class for Aircraft and Oxford-Pets and 10 for Stanford-Cars and CUB; long-tail validation holds out five test images per class.For long-tail datasets, the test set is split because some classes have too few unused training examples to form a separate validation set.Source: §6.3 · p. 14

7. Additional analysis

The ablations examine how fine-tuning duration and the amounts of synthetic and real data affect BOB’s performance relative to DataDream. BOB benefits more from longer fine-tuning, outperforms DataDream across the tested synthetic-image and real-image settings, and shows a narrowing advantage as real images increase.

7.1. Number of training epochs

  • From 200 to 400 epochs, BOB rises from 68.65% to 74.41%, while DataDream rises from 66.49% to 68.87%.Both methods peak at 1,000 epochs and decline at 2,000; BOB outperforms DataDream at every reported checkpoint.Source: §7.1 · p. 14, Figure 6 · p. 14

7.2. Number of synthetic images

  • Increasing synthetic images per class from 100 to 200–500 improves overall performance, and BOB outperforms DataDream at every tested amount.The ablation uses 10-shot Aircraft with an ImageNet-pretrained ResNet-50 and reduces epochs as image counts rise to hold gradient updates constant.Source: §7.2 · p. 14, Figure 7 · p. 14

7.3. Number of real images

  • With 1 real image per class, BOB reaches 25.18% accuracy versus DataDream’s 18.3%; with 16, the results are 80.53% and 78.91%, respectively.BOB outperforms DataDream at every tested real-image count, but the performance gap decreases as the count grows.Source: §7.3 · p. 14, Figure 8 · p. 14

7.4. Attributes ablations

The attribute ablations find similar performance for coupled and independent background–pose sampling, with performance increasing as more attributes are added. Separate runs with different few-shot images also consistently favor BOB over DataDream.

7.4. Attributes ablations

  • With background and pose, coupled sampling yields 60.31% versus 60.16% for independent sampling.The attribute ablation is on 5-shot Aircraft with an ImageNet-pretrained backbone.Source: §7.4 · p. 15, Table 7 · p. 15
  • Performance progressively increases as attributes are added, including the ablation that considers color or excludes background and pose.Color is an example attribute applicable to Aircraft but not CUB.Source: §7.4 · p. 15, Table 7 · p. 15

7.5. Stability across different seeds

  • Across three additional 5-shot Aircraft runs with different few-shot images, BOB outperforms DataDream each time.The runs use an ImageNet-pretrained backbone.Source: §7.5 · p. 15

7.6. Class differences in real vs. synthetic distribution

The classwise FID comparison favors BOB over DataDream for most classes, with its largest reported FID increase below five. Additional Car, CUB, and Pets examples are described as more diverse and realistic with BOB.

7.6. Class differences in real vs. synthetic distribution

  • For 91 of 100 classes, BOB has lower FID than DataDream; 29 classes see decreases greater than 5, while every increase is less than 5.The largest FID increase is 3.44.Source: §7.6 · p. 15, Figure 9 · p. 15
  • The classwise FID histogram compares DataDream and BOB across classes, making the frequency of lower or higher BOB FID the key comparison.The reported comparison covers 100 classes.Source: Figure 9 · p. 15, §7.6 · p. 15

7.7. Qualitative examples from Car, CUB, and Pets

  • Additional Car, CUB, and Pets examples show BOB producing more diverse and realistic synthetic images.The visualizations show the Acura Integra Type R 2001, American Goldfinch, and Abyssinian across different sources.Source: §7.7 · p. 15, Figure 10 · p. 15, Figure 11 · p. 16

7.8. Significant overlaps in the Pets dataset

On Pets, synthetic-data fine-tuning adds little for CLIP and ImageNet-pretrained ResNet-50, whose existing class recognition is strong. The analysis finds ImageNet classes for 25 of the 37 Pets classes.

7.8. Significant overlaps in the Pets dataset

  • Twenty-five of the 37 Pets classes have a corresponding ImageNet class.The table lists Pets classes alongside ImageNet IDs, using a dash when a class is absent.Source: §7.8 · p. 16, Table 9 · p. 16
  • For CLIP on Pets, all methods finish within 1% of fine-tuning on real data, while zero-shot accuracy is already 91%.The paper attributes the limited additional impact of synthetic data to CLIP’s strong existing classification capability.Source: §7.8 · p. 16
  • The paper cautions that a backbone with very high classification capability is not a good measure of the classification signal in synthetic data.The analysis applies this concern to the CLIP and ImageNet-pretrained settings on Pets.Source: §7.8 · p. 16
Loading 2510.24078v2…