Source-linked AI summary
Style Augmentation: Data Augmentation via Style Randomization
Philip T. Jackson, Amir Atapour-Abarghouei, Stephen Bonner, Toby Breckon, Boguslaw Obara
TL;DR
The paper addresses limited robustness of CNNs to domain shift and the expense of obtaining annotated data. It introduces style augmentation through randomized style transfer and reports improved generalization across classification, regression, and domain-transfer tasks.
Problem
CNNs rely heavily on expensive annotated data, while models trained on one domain often generalize poorly to other datasets despite the same task.
Method
Style augmentation randomizes image texture, contrast, and color through style transfer while preserving shape and semantic content.
Results
Style augmentation improves test accuracy and robustness across several vision tasks, particularly under domain shift, and can improve performance when combined with traditional augmentations.
Takeaways & Limitations
Data augmentation can reduce the need for domain adaptation by improving generalization without requiring information about a specific target domain.
Takeaways & Limitations
Results should not be compared directly with domain adaptation methods because style augmentation is domain agnostic, whereas domain adaptation uses information about a specific target domain.
Abstract
from arXiv · showhide
We introduce style augmentation, a new form of data augmentation based on random style transfer, for improving the robustness of convolutional neural networks (CNN) over both classification and regression based tasks. During training, our style augmentation randomizes texture, contrast and color, while preserving shape and semantic content. This is accomplished by adapting an arbitrary style transfer network to perform style randomization, by sampling input style embeddings from a multivariate normal distribution instead of inferring them from a style image. In addition to standard classification experiments, we investigate the effect of style augmentation (and data augmentation generally) on domain transfer tasks. We find that data augmentation significantly improves robustness to domain shift, and can be used as a simple, domain agnostic alternative to domain adaptation. Comparing style augmentation against a mix of seven traditional augmentation techniques, we find that it can be readily combined with them to improve network performance. We validate the efficacy of our technique with domain transfer experiments in classification and monocular depth estimation, illustrating consistent improvements in generalization.
1 Introduction
The paper proposes style augmentation, which randomizes image style while preserving semantic content, to improve robustness and generalization across tasks and domains.
- Data augmentation teaches models invariance to label-preserving transformations, but standard methods provide limited variation in color, texture, and illumination.The introduction contrasts geometric and intensity-based transforms with more complex low-level visual changes.
- Randomizing low-level visual features may improve generalization because neural networks can overfit to subtle distribution differences.Prior work supports this idea through randomized synthetic-scene appearance and image-to-image domain transformations.
- Style augmentation uses style transfer to randomize texture, color, and contrast while preserving image geometry.The approach applies randomized style transfer to arbitrary training images.
- The paper evaluates style augmentation on standard classification and domain transfer tasks, including models trained for unseen target domains.The authors also examine whether augmentation can implicitly improve domain transfer performance.
- Style randomization is implemented by sampling style embeddings to generate semantically valid images with random styles.This randomizes the action of a style transfer network rather than relying on a fixed target style image.
- Experiments use multiple tasks, architectures, and hyperparameter searches, and the implementation is released as a PyTorch augmentation package.The package is intended for deep learning practitioners.
2 Related Work
The related work frames domain shift and low-level visual bias as obstacles to generalization, then positions style augmentation as a domain-agnostic alternative to target-specific adaptation.
- 2.1 Domain Bias: Domain shift occurs when models trained on one data distribution generalize poorly to datasets from other domains.Differences in camera pose, illumination, lens properties, backgrounds, and distractors can create such shifts.
- 2.1 Domain Bias: Transfer learning and domain adaptation address domain shift, but adaptation typically targets a specific domain and may require additional training procedures.Transfer learning also imposes architectural and fine-tuning requirements.
- 2.1 Domain Bias: Compared with target-specific domain adaptation, source-domain augmentation can improve robustness across many potential target domains without target data.The paper presents this as a broader but domain-agnostic generalization strategy.
- 2.2 Style Transfer: Style transfer represents style through Gram-matrix feature correlations and content through high-level convolutional features.A joint objective combines style and content losses, traditionally using a pretrained loss network.
- 2.2 Style Transfer: Neural style-transfer networks approximate optimization-based style transfer with a single forward pass, enabling more efficient image transformation.Ghiasi et al. extend conditional normalization to arbitrary styles predicted from a style image.
- 2.2 Style Transfer: The paper uses an arbitrary-style transfer method, while acknowledging that augmentation quality depends on the chosen style-transfer approach.This dependence is an explicit scope boundary of the method.
- 2.3 Data Augmentation: Data augmentation introduces label-preserving transformations to teach invariance and improve neural-network generalization.Examples include cropping, mirroring, elastic distortions, dropout-like input perturbations, and random erasing.
- 2.3 Data Augmentation: CNNs can rely more on texture than shape, and randomized textures can encourage reliance on shape instead.This provides context for perturbing low-level appearance while retaining semantic structure.
3 Proposed Approach
The proposed approach adapts a fast, flexible arbitrary style-transfer network for augmentation by sampling style embeddings and interpolating them with the input image’s style. Conditional instance normalization uses these embeddings to control the transformer’s output style and augmentation strength.
- Style Transfer Pipeline: The method selects Ghiasi et al.’s style-transfer network because it is fast, flexible, and capable of applying broad styles.These properties are required for a practical augmentation technique.
- Randomization Procedure: Style embeddings are sampled directly from a multivariate normal distribution matched to embeddings from the Painter By Numbers dataset.This replaces the style predictor’s use of an explicit style image while simulating its embedding distribution.
- Style Transfer Pipeline: Conditional instance normalization shifts and rescales activation channels according to the style embedding, conditioning the transformer output on content and style.The style-dependent parameters are applied in convolutional layers except the first three.
- Randomization Procedure: The randomly sampled style embedding is linearly interpolated with the input image’s style embedding to control augmentation strength.Using the input style embedding leaves the image mostly unchanged, while varying α changes the transformer output style.
- Randomization Procedure: The random embedding depends on the input content image through the style predictor’s embedding P(c), together with the sampled distribution’s mean and covariance.The covariance matrix is defined from style-image embeddings P(s).
4 Experimental Results
Experiments across classification, cross-domain classification, and monocular depth estimation test style augmentation’s hyperparameters, combinations with traditional augmentation, and robustness to domain shift. Results generally favor combining style and traditional augmentation, while style augmentation also improves cross-domain and depth-estimation generalization.
- Experimental scope: The evaluation covers STL-10 classification, Office cross-domain classification, and monocular depth estimation on KITTI-related data.Experiments also search the augmentation ratio and style-transfer strength α.
- Hyperparameter search: A 2:1 unaugmented-to-augmented ratio and α = 0.5 appear optimal in the STL-10 hyperparameter searches.The ratio corresponds to an augmentation probability of 0.5; each experiment was repeated four times.
- Image classification: 8.5% improvement is achieved when style augmentation is combined with seven traditional augmentations on STL-10.Style augmentation alone also converges faster and reaches higher final accuracy than the unaugmented baseline.
- Image classification: 80.8% final test accuracy is reached on STL-10 using supervised training with strong augmentation and no unlabeled data.The authors report that this surpasses the reported state of the art.
- Cross-domain classification: On Office, models train on two domains and test on the third across four architectures and four augmentation settings.The settings are none, traditional, style, and both traditional plus style augmentation.
- Cross-domain classification: The combined style-and-traditional setting consistently achieves the highest final accuracy and fastest convergence on Office.Style augmentation alone can outperform all seven traditional techniques in some cases, particularly with InceptionV3.
- Ablation: Style augmentation yields accuracy gains at least 4% higher than color jitter on the Office ablation.The comparison uses InceptionV3 test accuracies.
- Monocular depth estimation: Models trained with style augmentation generalize better than traditionally augmented models for monocular depth estimation.Qualitative results also indicate sharper depth outputs with fewer artefacts.
5 Discussion
The discussion links style augmentation to texture-dependent CNN representations and examines how removing texture-label correlations affects performance across datasets. Its benefits are therefore dataset-dependent rather than uniformly positive.
- Texture bias: CNNs trained on ImageNet can rely more on texture than shape, supporting texture overfitting as a source of domain bias.The authors connect this prior finding to style augmentation as a practical way to combat domain bias.
- Dataset dependence: Style augmentation worsens ImageNet accuracy because it removes texture-label correlations that CNNs can exploit.The authors report that this behavior conforms to their texture-reliance hypothesis.
- Dataset dependence: Style augmentation moderately improves STL-10 validation accuracy, indicating that texture-label correlations differ across image-classification datasets.The contrast with ImageNet suggests that augmentation effects depend on dataset characteristics.
6 Conclusion
Style augmentation perturbs image color and texture while preserving shape and semantic content, improving CNN robustness, especially under domain shift. The findings also provide evidence linking CNN texture reliance to domain bias and support style augmentation as a practical way to reduce overfitting.
- Style augmentation perturbs image color and texture while preserving shape and semantic content to improve downstream CNN robustness.The method is designed as image-based data augmentation driven by style transfer.
- The approach yields significant improvements in test accuracy across several computer vision tasks, particularly under domain shift.
- The experiments provide evidence that CNN texture reliance contributes to domain bias and that style augmentation can mitigate domain bias and reduce overfitting.