PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning-State-Aware Dynamic Generative Data Augmentation on Small-Scale Datasets

Ting Xiang, Chenxi Deng, Jinhui Zhao, Bingting Jiang, Ke Zhang, Changjian Chen, Zhuo Tang

arXiv:2608.18907v1cs.CVcs.AI

TL;DR

Small-scale image classification lacks sufficient data, while existing dynamic generative augmentation struggles with sample-specific strengths and region-specific generation. LSADA adapts augmentation to each sample’s learning state and image regions, with experiments demonstrating effectiveness on natural and medical image classification tasks.

  • Problem

    Existing dynamic generative augmentation struggles to determine sample-specific strengths and adapt generation across image regions while balancing diversity and class semantics.

  • Method

    LSADA maps each sample’s loss and loss-decrease rate to augmentation strength, then fuses region-specific transformations and diffusion generation.

  • Results

    Experiments demonstrate LSADA’s effectiveness on natural and medical image classification tasks.

  • Takeaways & Limitations

    LSADA provides a learning-state-aware and region-specific augmentation approach for small-scale image classification.

  • Takeaways & Limitations

    The paper leaves extending LSADA to object detection, semantic segmentation, other generative models, and additional data modalities for future research.

Abstract

from arXiv · show

Small-scale image classification is often limited by the scarcity of training data. Generative data augmentation (GDA) based on pretrained generative models has emerged as an effective solution. However, existing methods rely on task-agnostic augmentation strategies that overlook downstream model needs. Although recent dynamic GDA methods incorporate model feedback to guide augmentation, they still struggle to reliably determine sample-specific augmentation strengths and adapt augmentation strategies to different image regions while balancing image diversity and class semantics. To address these issues, we propose learning-state-aware dynamic generative data augmentation (LSADA). Specifically, LSADA constructs a learning state for each sample based on its current loss and loss-decrease rate, which is then mapped to a sample-specific augmentation strength. Furthermore, LSADA introduces a decoupled data augmentation and diffusion fusion strategy that applies strength-controlled transformations to class-relevant regions and generates diverse class-irrelevant regions, progressively fusing them to improve image diversity while preserving class semantics. Experiments on nine public datasets show that LSADA outperforms the existing SOTA dynamic GDA method by an average of 4.5% on six natural image datasets and 2.5% on three medical image datasets.

1 Introduction

Small-scale datasets limit deep learning because collecting and annotating sufficient data is costly, motivating generative data augmentation. LSADA addresses unresolved sample- and region-specific augmentation needs by adapting augmentation strength to learning state and decoupling augmentation from diffusion-based generation.

  • Motivation: Small-scale data scenarios are constrained by the cost and time required to collect and manually annotate sufficient training images.Generative data augmentation uses pretrained generative models to generate diverse training images as a potential solution.
  • Challenges: Dynamic GDA uses downstream model feedback to select samples for augmentation but struggles with sample-specific strengths and region-specific requirements.Existing methods may apply uniform augmentation across entire images, overlooking different needs across image regions.
  • LSADA: LSADA characterizes each sample’s learning state using current loss and loss-decrease rate, then maps it to a sample-specific augmentation strength.Larger learning-state values receive weaker augmentation to preserve semantic features, while smaller values receive stronger augmentation to improve diversity.
  • LSADA: LSADA decouples data augmentation from diffusion fusion to improve generated-image diversity while preserving class semantics.The method’s generation strategy is designed to address region-specific augmentation requirements.

2 Related Work

Prior data augmentation methods comprise rule-based and generative approaches, with generative augmentation further divided into static and dynamic methods according to downstream-model feedback. LSADA addresses limitations in existing dynamic GDA by estimating sample learning states to control augmentation strength and decoupling region-specific augmentation.

  • Rule-Based Data Augmentation: Data augmentation methods are broadly categorized as rule-based augmentation and generative data augmentation.Rule-based methods apply predefined or sample-specific transformations, while generative methods use pretrained generative models.
  • Generative Data Augmentation: Static GDA generates samples independently of downstream training through latent perturbation, prompt optimization, or image-region editing.Its lack of downstream dependence may produce samples that are not fully beneficial to the classifier.
  • Generative Data Augmentation: Dynamic GDA uses downstream-model feedback during training to optimize image-to-image generation hyperparameters, including classification results, uncertainty, utility, and loss.This feedback controls the augmentation process rather than generating all samples before classifier training.
  • Generative Data Augmentation: LSADA estimates each sample’s learning state from current loss and loss-decrease rate, maps it to augmentation strength, and separately augments class-relevant and class-irrelevant regions.These mechanisms address existing dynamic GDA limitations in sample-specific strength selection and uniform augmentation.

3 Problem Formulation

Dynamic GDA updates generated data in response to downstream-classifier feedback at scheduled generation epochs. The resulting generated samples are combined with original and previously generated data to train the classifier using empirical classification loss.

  • Dynamic GDA formulation: Dynamic GDA adapts image generation to feedback from the downstream classifier, unlike static GDA.The classifier is represented as fθt: X → R^c at epoch t.
  • Dynamic GDA formulation: Generation occurs every k epochs, producing generation epochs Tg = {k, 2k, . . . , Rk}, where R = ⌊T/k⌋.T denotes the total augmentation training epochs, and R is the number of generation rounds.
  • Dynamic GDA formulation: At each generation epoch, classifier feedback determines each sample’s augmentation strength, and G generates mt samples per original sample.The value mt is specified by the generation ratio.
  • Dynamic GDA formulation: Generated data are combined with original and previously generated sets to update the classifier, which minimizes empirical loss over both data types.The classification loss L may be cross-entropy.

4 Method

LSADA estimates each sample’s learning state from current loss and loss-decrease rate, maps it to an augmentation strength, and decouples class-relevant transformations from class-irrelevant diffusion generation. Its fusion process preserves class semantics while generating diverse backgrounds through masked latent-space denoising.

  • Framework: LSADA comprises sample learning-state estimation and decoupled data augmentation with diffusion fusion generation.The learning state uses each sample’s current loss and loss-decrease rate, reflecting learning difficulty and recent progress.
  • Strength-controlled augmentation: LSADA maps each sample’s learning state to a strength controlling rule-based transformations applied to class-relevant regions.Class-relevant masks come from segmentation models when regions are defined, or class activation maps otherwise.
  • Sample learning-state estimation: The learning state combines current loss and loss-decrease rate, with larger values indicating harder-to-learn samples.The two quantities are combined using β > 0, which balances current learning difficulty and recent learning progress.
  • Decoupled augmentation: The method uses diverse background prompts generated by an LLM to guide generation of class-irrelevant regions rather than extracting them explicitly.This decouples class-relevant augmentation from class-irrelevant background generation to improve diversity while preserving semantics.
  • Diffusion fusion generation: Diffusion fusion encodes the augmented relevant region, performs masked latent-space denoising conditioned on a background prompt, and injects inverted relevant latents to preserve semantics.After masked injection across the fusion interval, reverse denoising continues without injection before decoding the final latent into an image.

5 Experiments · 5.1 Experimental Settings · 5.2 Overall Performance

LSADA is evaluated on nine small-scale natural and medical image classification datasets against rule-based, static generative, and dynamic generative augmentation baselines. It achieves higher accuracy, improves augmentation efficiency, and generalizes across downstream architectures under the reported experimental settings.

  • 5.1.1 Datasets.: Experiments cover nine public small-scale image classification datasets spanning coarse-grained, fine-grained, texture, and medical image tasks.Natural datasets include Caltech-101, CIFAR100-Subset, Cars, Flowers, Pets, and DTD; medical datasets are drawn from MedMNIST.
  • 5.1.2 Baselines.: LSADA is compared with rule-based methods, the static generative method GIF, and dynamic generative methods DisCL and ActGen.The rule-based baselines include CutOut, RandAugment, TrivialAugment, TeachAugment, MADAug, and EntAugment.
  • 5.1.3 Implementation details.: All datasets use ResNet-50 trained for 200 epochs, while SD 2.1 generates data during the first 100 epochs with updates every 5 epochs and generation ratio m = 5.SAM3 extracts class-relevant regions for five datasets, while CAM is used for the remaining datasets.
  • 5.2.1 DA effectiveness.: LSADA consistently outperforms rule-based and static generative augmentation methods and achieves higher accuracy than existing dynamic generative methods.The comparison is reported in Table 1 for ResNet-50 trained from scratch on original and generated images.
  • 5.2.1 DA effectiveness.: 4.5% average improvement on natural image datasets and 2.5% on medical image datasets is achieved by LSADA over ActGen, the SOTA dynamic GDA method.The results indicate that selecting samples solely from model predictions may provide insufficient guidance in small-scale settings.
  • 5.2.2 DA efficiency.: 10× improvement in data augmentation efficiency is demonstrated on DTD and Caltech-101, where LSADA at a 1× generation ratio exceeds ActGen at 10×.More generally, LSADA achieves competitive or higher accuracy than ActGen using fewer generated images across tested generation ratios.
  • 5.2.3 Generalization to various architectures.: LSADA consistently improves classification accuracy for ResNeXt-50, WideResNet-50, and MobileNet-V2 trained on 5×-generated Pets images.These images were generated using feedback from ResNet-50, testing architectural generalizability.

5.3 Ablation Study

Ablations show that LSADA benefits from combining current loss with loss-decrease rate, mapping learning state to sample-specific augmentation strength, and using decoupled augmentation–diffusion fusion. The generation strategy achieves the strongest balance among accuracy, diversity, semantic preservation, and natural image fusion.

  • Learning-state feedback: Combining current loss and loss-decrease rate produces the highest accuracy among the evaluated feedback signals, validating the proposed learning state.Compared variants include no feedback, ActGen’s prediction-based feedback, current loss only, and loss-decrease rate only.
  • Learning-state-to-strength mapping: The proposed mapping assigns weaker augmentation to larger learning-state values and stronger augmentation to smaller values, consistently outperforming the reversed mapping.This demonstrates the effectiveness of adapting augmentation strength to each sample’s learning state.
  • Generation strategy: The decoupled augmentation and diffusion fusion strategy achieves the highest accuracy, outperforming global image-to-image diffusion and direct pasting under otherwise unchanged components.The results indicate better balance between image diversity and class-semantic preservation than global editing or direct pasting.
  • Generation strategy: I2I produces diverse but semantically distorted images, Paste preserves semantics but appears unnatural, whereas LSADA preserves semantic consistency with more natural region-background fusion.I2I also has a much higher FID, while Paste has the lowest FID; an appropriate distribution gap may benefit downstream performance.

6 Conclusion

The paper proposes LSADA to address sample-specific augmentation-strength selection in dynamic GDA by modeling each sample’s learning state from its current loss and loss-decrease rate. Future work includes extending LSADA beyond image classification and integrating learning-state guidance with other generative models and data modalities.

  • 6 Conclusion: LSADA characterizes each sample’s learning state using its current loss and loss-decrease rate to adaptively determine augmentation strength.This addresses the difficulty of selecting sample-specific augmentation strengths in existing dynamic GDA methods.
  • 6 Conclusion: Future work could extend LSADA to object detection and semantic segmentation.The passage identifies these computer vision tasks as potential applications beyond the current setting.
  • 6 Conclusion: Future research could integrate learning-state guidance with other generative models and data modalities.The passage presents this integration as an additional avenue for investigation.

A Algorithm

LSADA alternates classifier optimization with periodic learning-state-aware image generation. It maps each sample’s loss dynamics to augmentation strength, generates augmented data, and returns the generated dataset with the final classifier.

  • Optimization loop: At each epoch, LSADA updates the classifier using the union of the original and previously generated datasets.The framework initializes the generated dataset as empty and iteratively expands it during training.
  • Learning-state adaptation: Every k epochs, LSADA computes each original sample’s current loss and loss-decrease rate, estimates its learning state, and maps normalized states to augmentation strengths.The update interval k controls when sample-specific learning states and augmentation strengths are recomputed.
  • Region-wise generation: For each sample, LSADA segments a class-relevant region and applies a rule-based transformation controlled by the computed augmentation strength.The algorithm extracts the region with a segmentation mask before transforming it.
  • Region-wise generation: LSADA generates prompts with an LLM, synthesizes mt images using the transformed region and generated inputs, and adds them to the generated dataset.After the augmentation loop, the classifier is trained on the union of original and generated data, and both outputs are returned.

B Datasets

The method is evaluated on nine public image-classification datasets spanning six natural-image and three medical-image datasets. The medical datasets use MedMNIST sources with tailored small-scale training splits.

  • Dataset composition: Nine publicly available datasets comprise six natural-image datasets and three medical-image datasets for evaluation.Dataset statistics are summarized in Table 7.
  • Natural image datasets: The natural-image datasets include Caltech 101, CIFAR100-Subset, Stanford Cars, Oxford 102 Flowers, Oxford-IIIT Pets, and DTD.CIFAR100-Subset samples 100 images per class from CIFAR100, totaling 10,000 images across 100 categories.
  • Medical image datasets: The medical datasets are BreastMNIST, PathMNIST, and OrganSMNIST from the MedMNIST benchmark, covering breast ultrasound, colon pathology, and abdominal CT images.BreastMNIST and PathMNIST use official validation splits for training, while OrganSMNIST retains its original training split.

C Implementation Details

LSADA trains the classifier for 200 epochs while generating augmented images during the first 100 epochs through 20 scheduled rounds. The implementation uses fixed diffusion cycles with dataset-specific late-stage masked injection settings.

  • Training and generation schedule: 200 classifier epochs include 100 augmentation epochs, with 20 generation rounds scheduled every 5 epochs from 5 through 100.The overall generation ratio is m = 5, corresponding to a target per-round ratio of 0.25.
  • Training and generation schedule: Approximately n/4 original samples are selected without replacement at each generation epoch, producing one generated image per selected sample and about 5n images overall.Individual originals need not be selected exactly five times; generated samples accumulate in Dg and are used with Do after each round.
  • Training and generation schedule: After epoch 100, generation stops, and training continues on Do ∪ Dg through epoch 200.This schedule separates the augmentation phase from the remaining classifier training.
  • Diffusion configuration: Each diffusion cycle uses N = 10 inference steps, injects noise from τ1 = 5 to τ2 = 7, applies masked injection at {7, 6, 5}, and repeats 10 times.Reverse denoising proceeds from τ2 to τ1 after noise injection.
  • Diffusion configuration: Late-stage masked injection uses Tinj = {4, 3} for five natural-image datasets and Tinj = {4} for CIFAR100-Subset, PathMNIST, BreastMNIST, and OrganSMNIST.At each selected timestep, the inverted class-relevant latent is injected into the class-relevant region.

D Visualization on Medical Datasets

Figure 7 visualizes medical-dataset samples generated by three strategies. I2I preserves main medical structures but changes global appearance and texture, while Paste and LSADA use shared CAM maps to preserve class-related regions.

  • I2I generally preserves the main medical structures but introduces noticeable changes in global appearance and texture, potentially causing semantic distortion.
  • Paste and LSADA use the same CAM maps to preserve class-related regions, but Paste directly blends preserved regions with generated backgrounds in pixel space.
  • Figure 7 presents medical-image samples produced by I2I, Paste, and LSADA.
Loading 2608.18907v1…