Implicit Semantic Data Augmentation for Deep Networks

Yulin Wang; Xuran Pan; Shiji Song; Hong Zhang; Cheng Wu; Gao Huang

Source-linked AI summary

Implicit Semantic Data Augmentation for Deep Networks

Yulin Wang, Xuran Pan, Shiji Song, Hong Zhang, Cheng Wu, Gao Huang

arXiv:1909.12220v5Sep 26, 2019cs.CVcs.LGstat.ML

TL;DR

Deep networks offer semantic directions in feature space, but conventional augmentation misses transformations such as background or texture changes, while generative approaches are costly. ISDA samples class-relevant directions from online covariance estimates and optimizes a closed-form robust cross-entropy upper bound instead of generating samples explicitly. Across CIFAR and ImageNet evaluations, it consistently improves generalization with negligible extra computational cost.

Problem
Conventional augmentation cannot perform semantic transformations, while existing generative semantic augmentation requires auxiliary models and substantial computation.
Method
ISDA samples semantic directions from zero-mean class-conditional covariance distributions and minimizes a closed-form upper bound on the augmented cross-entropy loss.
Results
ISDA consistently improves generalization across image-classification benchmarks, including CIFAR-10, CIFAR-100, and ImageNet.
Takeaways & Limitations
ISDA complements traditional augmentation and is compatible with deep networks without auxiliary models or noticeable extra computational cost.
Takeaways & Limitations
The method assumes that class-specific meaningful semantic transformations are well represented by the principal components of each class's feature covariance.

Abstract

from arXiv · show

In this paper, we propose a novel implicit semantic data augmentation (ISDA) approach to complement traditional augmentation techniques like flipping, translation or rotation. Our work is motivated by the intriguing property that deep networks are surprisingly good at linearizing features, such that certain directions in the deep feature space correspond to meaningful semantic transformations, e.g., adding sunglasses or changing backgrounds. As a consequence, translating training samples along many semantic directions in the feature space can effectively augment the dataset to improve generalization. To implement this idea effectively and efficiently, we first perform an online estimate of the covariance matrix of deep features for each class, which captures the intra-class semantic variations. Then random vectors are drawn from a zero-mean normal distribution with the estimated covariance to augment the training data in that class. Importantly, instead of augmenting the samples explicitly, we can directly minimize an upper bound of the expected cross-entropy (CE) loss on the augmented training set, leading to a highly efficient algorithm. In fact, we show that the proposed ISDA amounts to minimizing a novel robust CE loss, which adds negligible extra computational cost to a normal training procedure. Although being simple, ISDA consistently improves the generalization performance of popular deep models (ResNets and DenseNets) on a variety of datasets, e.g., CIFAR-10, CIFAR-100 and ImageNet. Code for reproducing our results is available at https://github.com/blackfeather-wang/ISDA-for-Deep-Networks.

1 Introduction

ISDA augments deep-network training by translating features along class-relevant semantic directions, avoiding auxiliary networks and explicit sample generation. It uses a closed-form robust loss and improves generalization across image-classification benchmarks.

Motivation: Semantic augmentation addresses transformations that conventional cropping, mirroring, rotation, and color jittering cannot perform, such as changing backgrounds or foreground texture.Generative approaches can provide such transformations but require computationally intensive training and inference.
Motivation: ISDA uses semantic directions in deep feature space to create class-preserving transformations, complementing conventional augmentation.The approach relies on feature linearization, where directions can represent changes such as adding glasses or altering backgrounds.
Method: Online class-conditional covariance estimates provide random semantic directions while balancing transformation effectiveness and computational efficiency.Vectors are sampled from a zero-mean normal distribution whose covariance captures intra-class feature variation.
Method: ISDA minimizes a closed-form upper bound on expected cross-entropy instead of explicitly generating augmented samples.This formulation avoids auxiliary networks and noticeable extra computational cost.
Results: ISDA consistently improves generalization for popular deep networks, especially with little training data and strong traditional augmentation.The paper reports extensive evaluations on competitive image-classification benchmarks.

2 Related Work

Related work covers conventional and automatic augmentation, implicit augmentation through marginalized corrupted features, semantic augmentation with generative models, and robust loss functions. ISDA connects these areas by targeting semantic feature variation without the computational burden of auxiliary generators.

Data Augmentation: Conventional augmentation enforces invariance through transformations such as flipping, mirroring, rotation, and color changes, while automatic methods search among candidate strategies.The related-work discussion also identifies marginalized corrupted features as implicit augmentation limited to simple linear models.
Semantic Transformations: Semantic augmentation applies class-identity-preserving changes such as backgrounds or visual angles, commonly using GANs or specialized deep structures.These approaches are described as effective but nontrivial to implement and computationally expensive.
Robust Loss Function: Robust-loss research includes Lq, focal, large-margin, contrastive, and center losses, addressing noise robustness, hard examples, margins, or feature compactness.ISDA is situated in this literature because the paper formulates it as a robust loss function.

3 Method

ISDA augments deep features along class-specific semantic directions sampled from online covariance estimates, then replaces explicit sample generation with optimization of an efficient robust-loss upper bound. The method is implemented by estimating covariances, computing the surrogate loss, and updating the network with SGD.

Semantic transformations: Deep feature translations can represent meaningful semantic transformations, motivating augmentation directly in feature space.Examples include directions associated with changes such as wearing glasses or viewing angle.
Semantic transformations: ISDA samples semantic directions from a zero-mean normal distribution whose class-conditional covariance is estimated online from feature statistics.The covariance is computed separately for each class and is intended to capture class-specific intra-class variations.
Semantic transformations: The augmented feature satisfies ˜a_i ∼ N(a_i, λΣ_yi), with λ controlling augmentation strength and increasing during training to reduce reliance on early covariance estimates.The schedule λ = (t/T)×λ0 gradually increases the augmentation coefficient.
Implicit semantic data augmentation: Explicitly replicating each feature M times enlarges the training set and becomes computationally inefficient when M is large.The method instead considers the expected cross-entropy loss over infinitely many possible augmented features.
Implicit semantic data augmentation: A closed-form upper bound of the expected cross-entropy provides a surrogate robust loss that can be minimized without explicitly generating augmented samples.The exact expectation is difficult to compute, while the upper bound is efficiently optimized within the training procedure.
Optimization: The training algorithm computes features and class covariances per mini-batch, evaluates the robust loss, and updates W, b, and Θ with SGD.When λ → 0, the robust loss reduces to the standard cross-entropy loss.

4 Experiments

Experiments evaluate ISDA across image-classification datasets, architectures, augmentation methods, competing approaches, visualizations, and ablations. ISDA consistently improves generalization, complements traditional augmentation, and benefits from class-specific covariance modeling.

Experimental setup: Experiments cover CIFAR-10, CIFAR-100, and ImageNet using multiple modern deep-network architectures and validation procedures.The study evaluates ResNet, SE-ResNet, Wide-ResNet, ResNeXt, and DenseNet models, with CIFAR validation data used to select λ0.
Main results: On ImageNet, ISDA reduces ResNet-50 Top-1 error by 1.1%, reaching 21.9% versus ResNet-101’s 21.7% with 43% fewer parameters.ResNet-101+ISDA also surpasses ResNet-152 with 26% fewer parameters, while DenseNets benefit less than ResNets.
Main results: On CIFAR-100, ISDA reduces test errors by about 1% for ResNet-32 and ResNet-110 and nearly 0.7% against competitive baselines for larger models.The larger models are Wide-ResNet-28-10 and ResNeXt-29, 8x64d.
Complementarity with augmentation: With AutoAugment on CIFAR-100, ISDA gains 1.34% for Shake-Shake (26, 2x112d) and 0.98% for Wide-ResNet-28-10.The reported gains are more significant than in standard augmentation settings, and training curves show stronger improvement after later learning-rate drops.
Comparison with other approaches: Against robust-loss and generator-based semantic-augmentation baselines, ISDA compares favorably and achieves ResNet-110 test errors of 6.23% on CIFAR-10 and 27.11% on CIFAR-100.Other robust losses obtain 6.38% and 27.85% on the corresponding datasets; GAN-based methods can degrade on CIFAR-100 and require generator-training overhead.
Visualization and ablation: Visualizations show ISDA altering backgrounds, visual angles, colors, car types, and skin colors, while ablations show that identity, shared, or diagonal covariance choices perform worse.The ablation results attribute these degradations to unsuitable directions or failure to model feature correlations.

5 Conclusion

ISDA is an efficient semantic augmentation algorithm that complements existing augmentation techniques through a robust loss compatible with any deep network using cross-entropy.

ISDA complements existing data augmentation techniques with an efficient implicit semantic data augmentation algorithm.
ISDA is formulated as a novel robust loss function compatible with any deep network trained using cross-entropy loss.
Extensive experiments on several competitive image classification datasets demonstrate ISDA’s effectiveness and efficiency.

A Implementation Details of ISDA.

During training, ISDA dynamically estimates class-specific feature statistics and computes gradients for its robust loss through backpropagation.

Covariance matrices are dynamically estimated during training for each class’s feature representations.The implementation passages identify these as estimates of class-wise feature means and covariance matrices across training steps.
The class-wise sample count tracks the total number of training samples from each class across the relevant mini-batch history.
The mini-batch-specific count records how many samples from a given class occur only in the current mini-batch.
Gradients of the robust ISDA loss are computed during backward propagation and passed through the network using the derivative with respect to activations.

B Training Details

Experiments use standard deep architectures and shared training configurations across CIFAR and ImageNet, with baselines and generator-based methods tuned for comparison.

CIFAR experiments train ResNet, SE-ResNet, Wide-ResNet, ResNeXt, and DenseNet models using SGD with Nesterov momentum.Specific CIFAR hyperparameters are reported in Table 6.
ImageNet models are trained for 300 epochs with cosine learning-rate annealing, a mini-batch size of 512, and model-specific λ0 values.λ0 is 1 for DenseNets, 7.5 for ResNets and ResNeXts, and 5 for ResNet-101.
Baseline comparisons use the same training configurations, with specified dropout, label-noise, focal-loss, Lq-loss, and center-loss settings.
The algorithm overview uses a fixed WGAN generator trained to produce fake images, while optimizing generator inputs for pixel-space and deep-feature consistency.
Generator-based augmentation methods use class-specific or class-conditional GAN designs with 100-dimensional standard-normal noise as input.

C Reversing Convolutional Networks

The paper reverses convolutional networks by optimizing inputs to a fixed generator so generated images match target deep features while retaining pixel-space consistency.

Extra visualization results are presented in Figure 5 to demonstrate semantic changes generated by ISDA.
A fixed pretrained generator is added because convolutional networks lack closed-form inverse functions, enabling image reconstruction from target features by input optimization.
The reconstruction procedure has two steps: first identify generator noise corresponding to a real image, then reconstruct an ISDA-augmented feature.
Step I: Step I optimizes normalized generator noise so the generated image matches the real image in both deep-feature and pixel spaces.The relative importance of the two objectives is controlled by η.
Step II: Step II searches for a new generator input corresponding to the ISDA-augmented feature, initialized from the noise found in Step I.
The visualization algorithm uses a ResNet-32 and standard gradient descent for 10000 iterations to solve the reconstruction equations.Learning rates are initialized separately for the two steps and reduced every 2500 iterations, with momentum 0.9 and L2 weight decay 1e-4.

D Extra Experimental Results

Figure 6 compares test-error curves for ISDA and state-of-the-art image-classification methods. ISDA consistently achieves the best generalization performance, with more evident test-error reductions on CIFAR-100.

ISDA consistently outperforms state-of-the-art image-classification methods across the situations shown in Figure 6.The comparison uses test-error curves.
ISDA achieves the best generalization performance in all situations presented.
ISDA decreases test errors more evidently on CIFAR-100, suggesting greater suitability for datasets with fewer samples.The passage states that this observation is consistent with results elsewhere in the paper.