PaperlayerView exact revision on arXiv

Source-linked AI summary

Regularizing Deep Networks with Semantic Data Augmentation

Yulin Wang, Gao Huang, Shiji Song, Xuran Pan, Yitong Xia, Cheng Wu

arXiv:2007.10538v5cs.CVcs.LG

TL;DR

Conventional augmentation lacks semantic diversity, while generative semantic augmentation is computationally costly. ISDA translates deep features along sampled semantic directions and replaces explicit augmentation with a closed-form robust loss, consistently improving generalization across classification and segmentation benchmarks with minimal overhead.

  • Problem

    Conventional augmentation uses low-level transformations with limited diversity, while semantic augmentation through generative models requires costly training, inference, and expanded-data processing.

  • Method

    ISDA samples class-conditional semantic directions in deep feature space and minimizes a closed-form upper bound on the expected augmented CE loss, extending to semi-supervised consistency training.

  • Results

    ISDA consistently improves generalization across supervised and semi-supervised image classification and Cityscapes semantic segmentation benchmarks.

  • Takeaways & Limitations

    ISDA complements existing augmentation and can be implemented with most deep models without auxiliary networks or noticeable extra computational cost.

  • Takeaways & Limitations

    The surrogate objective requires the upper bound of the expected loss to be sufficiently tight, and semantic feature transformations need not be explicitly reconstructed in pixel space.

Abstract

from arXiv · show

Data augmentation is widely known as a simple yet surprisingly effective technique for regularizing deep networks. Conventional data augmentation schemes, e.g., flipping, translation or rotation, are low-level, data-independent and class-agnostic operations, leading to limited diversity for augmented samples. To this end, we propose a novel semantic data augmentation algorithm to complement traditional approaches. The proposed method is inspired by the intriguing property that deep networks are effective in learning linearized features, i.e., certain directions in the deep feature space correspond to meaningful semantic transformations, e.g., changing the background or view angle of an object. Based on this observation, translating training samples along many such directions in the feature space can effectively augment the dataset for more diversity. To implement this idea, we first introduce a sampling based method to obtain semantically meaningful directions efficiently. Then, an upper bound of the expected cross-entropy (CE) loss on the augmented training set is derived by assuming the number of augmented samples goes to infinity, yielding a highly efficient algorithm. In fact, we show that the proposed implicit semantic data augmentation (ISDA) algorithm amounts to minimizing a novel robust CE loss, which adds minimal extra computational cost to a normal training procedure. In addition to supervised learning, ISDA can be applied to semi-supervised learning tasks under the consistency regularization framework, where ISDA amounts to minimizing the upper bound of the expected KL-divergence between the augmented features and the original features. Although being simple, ISDA consistently improves the generalization performance of popular deep models (e.g., ResNets and DenseNets) on a variety of datasets, i.e., CIFAR-10, CIFAR-100, SVHN, ImageNet, and Cityscapes.

1 INTRODUCTION

ISDA complements conventional pixel-space augmentation by generating semantic feature variations and optimizing their expected loss implicitly. It uses sampled class-conditional directions and a closed-form surrogate, extending to semi-supervised learning while improving performance across vision benchmarks.

  • Conventional pixel-space transformations offer limited semantic diversity, while class-preserving changes such as color or background are complementary augmentation strategies.
  • ISDA translates deep features along semantic directions, avoiding auxiliary networks and explicit generation of additional training samples.These directions represent class-preserving semantic variations such as changing appearance or background.
  • Class-conditional feature covariance estimates provide an efficient balance between meaningful semantic directions and the cost of human annotation.
  • A closed-form upper bound of the expected CE loss turns semantic augmentation into a robust surrogate loss with negligible additional computational overhead.
  • For semi-supervised learning, ISDA applies semantic consistency to unlabeled features through an upper bound on expected KL-divergence.
  • ISDA consistently improves generalization across supervised and semi-supervised classification and Cityscapes semantic segmentation benchmarks.

2 RELATED WORK

Related work covers conventional and automated augmentation, robust loss functions, semantic manipulation in deep feature spaces, uncertainty modeling, and semi-supervised learning. ISDA connects these areas by using semantic feature directions and a robust-loss formulation.

  • Classic augmentation enforces geometric invariance through transformations such as flipping, mirroring, rotation, cropping, and color changes.
  • Automatic and generative augmentation methods seek broader variation, but semantic generation can require costly model training and inference.
  • Robust loss function: Robust-loss research includes Lq, focal, margin-based, and contrastive formulations aimed at generalization, label robustness, or harder-example emphasis.
  • Semantic transformations via deep features: Deep feature interpolation and latent-variable models show that high-level representations can encode semantic abstractions manipulable through feature or latent-space transformations.
  • Uncertainty modeling: Uncertainty-modeling methods use probabilistic representations for ambiguous faces, outliers, label noise, and task uncertainty.
  • Deep semi-supervised learning: Semi-supervised learning leverages abundant unlabeled data alongside limited labeled data because precise annotations are expensive and time-consuming.

3 SEMANTIC TRANSFORMATIONS IN DEEP FEATURE SPACE

Deep networks organize semantic relationships in feature space, where selected directions correspond to meaningful image transformations. The paper uses this property to augment training directly in feature space without reconstructing transformed pixels.

  • Deep features capture semantic relationships through their spatial positions in the learned feature space.
  • Linear translation along suitable feature directions can realize transformations such as changing a car’s color or an image’s background.
  • The proposed method augments training examples by translating their deep features along many meaningful semantic directions.
  • Because augmented features can be used directly for training, explicit pixel-space reconstruction is unnecessary.

4 IMPLICIT SEMANTIC DATA AUGMENTATION (ISDA)

ISDA augments deep features along class-relevant semantic directions while avoiding the cost and incompleteness of annotation-based directions or the meaningless transformations produced by unrestricted random sampling. It estimates class-conditional feature covariance online and replaces explicit infinite-sample augmentation with an efficiently optimized robust loss.

  • Semantic Direction Sampling: ISDA uses deep-feature semantic directions to augment samples while preserving the main object's class identity.The method targets transformations such as changing visual angle or wearing glasses, while excluding class-inappropriate transformations.
  • Semantic Direction Sampling: Human annotation is precise but expensive and incomplete, whereas unrestricted random sampling is efficient but produces many meaningless transformations.The paper motivates covariance-guided sampling as a trade-off between these alternatives.
  • Semantic Direction Sampling: Class-conditional covariance matrices encode intra-class variation, enabling semantic directions to be sampled from a zero-mean normal distribution tailored to each class.The covariance is estimated online by aggregating feature statistics across mini-batches.
  • Semantic Direction Sampling: The augmented feature is formed by translating the original feature along a random direction sampled from N(0, λΣ_yi), with λ scheduled from zero to λ0 during training.The schedule reduces the influence of poorly estimated covariances early in training.
  • Upper Bound of the Expected Loss: Instead of explicitly generating M augmented samples, ISDA considers the M →∞ expected CE loss and optimizes an easy-to-compute upper bound as a robust surrogate loss.When λ →0, the surrogate reduces to the standard CE loss.
  • Complexity of ISDA: For ResNet-110 on CIFAR, ISDA's extra computation is up to three orders of magnitude smaller than the network's total computation cost.The added per-sample costs are O(D^2) for covariance updates and O(C ×D^2) for the upper-bound computation.

5 ISDA FOR DEEP SEMI-SUPERVISED LEARNING

ISDA extends semantic augmentation to semi-supervised learning by preserving predictions under feature transformations, while replacing explicit sampling with efficient surrogate losses.

  • The method augments unlabeled deep features and minimizes the KL-divergence between predictions for augmented and original features.This consistency assumption follows because ISDA preserves class identity.
  • A closed-form upper bound of the expected KL-divergence serves as an efficient surrogate, avoiding naive averaging over many augmented samples.The naive approach becomes inefficient as the augmented feature set grows, motivating the limit M →∞.
  • For labeled samples, the algorithm minimizes the upper bound used by supervised ISDA, while unlabeled samples contribute through semantic consistency.The combined loss appends the relevant regularization term with predefined coefficients.
  • Semi-supervised ISDA amounts to minimizing a novel robust loss and can be integrated efficiently with existing deep semi-supervised learning algorithms.

6 EXPERIMENTS

Experiments evaluate ISDA across supervised and semi-supervised classification, semantic segmentation, feature representations, and upper-bound tightness. ISDA generally improves performance with low computational overhead, while its benefits vary with architecture and covariance modeling.

  • Supervised image classification: On CIFAR-100, ISDA reduces test errors by about 1% for smaller ResNets and outperforms competitive baselines by nearly 0.7% for larger models.The larger-model comparison is reported for Wide-ResNet-28-10 and ResNeXt-29, 8x64d.
  • Complementing explicit augmentation: ISDA complements explicit augmentation, yielding 1.34% and 0.98% gains with AutoAugment on CIFAR-100 for Shake-Shake and Wide-ResNet-28-10, respectively.The paper suggests non-semantic augmentation may improve feature representations, making semantic transformations more reliable.
  • Semi-supervised image classification: ISDA improves semi-supervised methods, reducing VAT test error by 1.38% with 4,000 labeled CIFAR-10 samples and by 3.74% and 4.08% in lower-label settings.The latter gains are reported for CIFAR-10 with 1,000 labeled samples and CIFAR-100 with 10,000 labeled samples, respectively.
  • Semantic segmentation: ISDA improves Cityscapes segmentation by nearly 1% mIoU for both baselines, with about a 6% increase in training time.The method augments each pixel during training and approximates covariance matrices by their diagonals to save GPU memory.
  • Analysis and efficiency: ISDA produces tighter feature clusters, introduces at most 4.25% theoretical overhead, and yields a tight upper bound on CIFAR-10 and CIFAR-100.Empirical training-time increases are 5% to 7% on ImageNet and 1% to 12% on CIFAR; explicit augmentation approaches ISDA as M grows.

7 CONCLUSION

ISDA efficiently complements conventional augmentation by using a robust loss formulation compatible with deep networks and semi-supervised consistency training. Experiments across vision benchmarks demonstrate its effectiveness and efficiency.

  • ISDA complements existing data augmentation techniques with an efficient implicit semantic data augmentation algorithm.The approach differs from generative methods by avoiding explicit semantically transformed samples.
  • ISDA is formulated as a robust loss function compatible with any deep network using softmax cross-entropy loss.
  • ISDA can be implemented efficiently in semi-supervised learning through semantic consistency training.
  • Extensive experiments on several competitive vision benchmarks demonstrate the algorithm's effectiveness and efficiency.

APPENDIX A TRAINING DETAILS FOR SUPERVISED LEARNING

The supervised-learning appendix specifies CIFAR training configurations and identifies Table 9 as the source of detailed hyper-parameters.

  • CIFAR experiments use ResNet, SE-ResNet, Wide-ResNet, ResNeXt, and DenseNet models.
  • All CIFAR models are trained with SGD using Nesterov momentum.
  • Table 9 presents the specific training configurations used for CIFAR experiments.The table defines training settings, including the learning-rate notation.

TRAINING DETAILS FOR SEMI-SUPERVISED LEARNING

The semi-supervised-learning appendix uses CNN-13 with a specified optimization schedule, while Table 10 documents its architecture.

  • Semi-supervised experiments use the CNN-13 network, a widely used architecture for semi-supervised learning.
  • CNN-13 is trained for 400 epochs with SGD and Nesterov momentum.
  • Training uses an initial learning rate of 0.1 annealed to zero with cosine annealing.
  • Table 10 shows the architecture of CNN-13.

HYPER-PARAMETER SELECTION FOR BASELINES

Baseline comparisons use specified regularization, noise, generator, and hyper-parameter settings, with semi-supervised implementations following the original papers and detection and segmentation results reported on COCO.

  • Baseline dropout is set to 0.3 when absent from the basic supervised model.
  • Disturb-label noise rates are set to 0.05 or 0.1 depending on the model and CIFAR dataset.
  • Generator-based methods use published GAN structures, with class-conditional generation configurations varying by method.
  • Semi-supervised baselines use the original papers' hyper-parameter settings and search policies.The implementations achieve the same or sometimes better results under those settings.
  • COCO detection and instance-segmentation comparisons report AP-based metrics for models with and without ISDA.The tables cover detection backbones and Mask-RCNN with different backbones.

APPENDIX D RESULTS ON COCO

This appendix evaluates ISDA-enhanced ResNet-50 backbones on object detection and instance segmentation using MS COCO. It compares Faster-RCNN, Mask-RCNN, and Cascade-RCNN under MMDetection's default configuration with FPN backbones.

  • ResNet-50 models reported in Table 1 serve as backbones for MS COCO object detection and instance segmentation.
  • The evaluation includes Faster-RCNN, Mask-RCNN, and Cascade-RCNN.
  • All three methods use MMDetection's default configuration with FPN backbones, with results presented in Tables 11 and 12.

APPENDIX E VISUALIZATION OF AUGMENTED SAMPLES

The appendix reconstructs ISDA-augmented deep features as images by optimizing the inputs of a fixed generator while preserving pixel and feature consistency. The visualizations show semantic changes and indicate that augmented ImageNet samples are not simply memorized training examples.

  • Reverse mapping: The reverse mapping algorithm fixes a pretrained generator and adjusts its input to reconstruct images corresponding to augmented deep features.Direct pixel-space optimization is described as ineffective for reconstructing semantic changes.
  • Reverse mapping: Step I finds a generator noise variable corresponding to each real image by matching both deep features and pixels.The relative weighting of the feature-space and pixel-space objectives is controlled by η.
  • Reverse mapping: Step II augments the original feature with ISDA and searches for a new generator input that reconstructs the augmented feature.The search starts from the noise variable found in Step I.
  • Visual results: ISDA visualizations alter semantics unrelated to class identity, including backgrounds, skin colors, and car types.
  • Visual results: ImageNet augmented samples are visually distinct from their Top-5 pixel-space nearest neighbors, suggesting they are not produced by memorizing training data.
Loading 2007.10538v5…