Data augmentation instead of explicit regularization

Alex Hernández-García; Peter König

Source-linked AI summary

Data augmentation instead of explicit regularization

Alex Hernández-García, Peter König

arXiv:1806.03852v5Jun 11, 2018cs.CV

TL;DR

The paper examines whether explicit regularization is necessary when deep networks already contain implicit regularization, and formalizes the distinction between the two. It contrasts data augmentation with weight decay and dropout across visual-recognition models, finding that augmentation alone matches or outperforms their combination. The authors therefore argue for emphasizing data augmentation while avoiding unnecessary tuning of explicit regularizers.

Problem
The paper asks how explicit and implicit regularization interact in deep networks and whether commonly used weight decay and dropout add generalization benefits beyond existing inductive biases.
Method
The paper formalizes explicit and implicit regularization and experimentally compares data augmentation alone with data augmentation combined with weight decay and dropout across architectures and benchmarks.
Results
Data augmentation alone achieved better accuracy than augmentation plus explicit regularization in 9/16 cases, with a positive difference at 95% confidence and P value = 0.0220.
Takeaways & Limitations
The findings support using data augmentation as a primary regularization strategy and reconsidering weight decay and dropout when sufficient implicit regularization is present.
Takeaways & Limitations
The experiments used a limited set of three architectures and three datasets, and retained original hyperparameters even when training without explicit regularization.

Abstract

from arXiv · show

Contrary to most machine learning models, modern deep artificial neural networks typically include multiple components that contribute to regularization. Despite the fact that some (explicit) regularization techniques, such as weight decay and dropout, require costly fine-tuning of sensitive hyperparameters, the interplay between them and other elements that provide implicit regularization is not well understood yet. Shedding light upon these interactions is key to efficiently using computational resources and may contribute to solving the puzzle of generalization in deep learning. Here, we first provide formal definitions of explicit and implicit regularization that help understand essential differences between techniques. Second, we contrast data augmentation with weight decay and dropout. Our results show that visual object categorization models trained with data augmentation alone achieve the same performance or higher than models trained also with weight decay and dropout, as is common practice. We conclude that the contribution on generalization of weight decay and dropout is not only superfluous when sufficient implicit regularization is provided, but also such techniques can dramatically deteriorate the performance if the hyperparameters are not carefully tuned for the architecture and data set. In contrast, data augmentation systematically provides large generalization gains and does not require hyperparameter re-tuning. In view of our results, we suggest to optimize neural networks without weight decay and dropout to save computational resources, hence carbon emissions, and focus more on data augmentation and other inductive biases to improve performance and robustness.

1 Introduction

Deep networks combine explicit and implicit sources of regularization, but their interactions and contributions to generalization remain poorly understood. The paper investigates these interactions and contrasts data augmentation with weight decay and dropout.

Deep learning uses multiple regularization sources, including explicit techniques and implicit inductive biases from network components and training procedures.Examples of implicit sources include convolutional layers, batch normalization, and data augmentation.
Weight decay, dropout, and data augmentation are commonly combined, implicitly assuming that each contributes additively to better generalization.The paper identifies the interaction between these techniques as an unresolved issue.
Explicit regularizers such as weight decay and dropout require hyperparameter tuning and constrain representational capacity, affecting training costs and model capacity.The motivation is to understand whether these costs are justified when other regularizing elements are already present.
The paper defines explicit and implicit regularization, theoretically contrasts data augmentation with explicit regularization, and empirically evaluates them across benchmarks.It also studies adaptation to fewer examples and architectural changes.

2 Explicit and implicit regularization

The paper addresses ambiguity in the literature by defining explicit and implicit regularization through representational and effective capacity. It classifies data augmentation as implicit regularization and distinguishes it from capacity-reducing techniques such as weight decay and dropout.

The literature uses explicit and implicit regularization inconsistently, motivating formal definitions to clarify their meanings.The paper notes that existing taxonomies do not provide generally accepted formal definitions.
Explicit regularization reduces a model class’s representational capacity, whereas implicit regularization reduces generalization error or overfitting without reducing representational capacity.Implicit regularization may instead affect effective capacity through the model, optimization algorithm, or hyperparameters.
Weight decay constrains representational capacity through an L2 penalty, and dropout reduces the hypothesis set by stochastically deactivating units.Both are presented as common examples of explicit regularization.
Implicit regularization is described as an effect that can arise from diverse elements, including SGD, batch normalization, and early stopping.These elements can influence effective capacity without directly restricting the model class.
Data augmentation is classified as implicit because it does not reduce model representational capacity, despite improving generalization through an inductive bias.This classification differs from treatments that place data augmentation alongside weight decay and dropout.

3 Theoretical insights

The theoretical analysis explains why data augmentation may provide stronger generalization support than generic capacity constraints. It connects augmentation to increased effective sample availability and domain knowledge, while noting that its formal effect is difficult to analyze.

The paper hypothesizes that data augmentation’s inductive bias can overshadow the generalization gains from weight decay and dropout.Empirical evidence is presented separately, while this section develops statistical-learning-theory insights supporting the hypothesis.
Generalization bounds depend strongly on the number of training examples N, making sample-size growth relevant to generalization guarantees.The paper uses this dependence to motivate the role of data augmentation.
Perceptually plausible augmentation uses domain knowledge and visual-perception priors to create new examples, potentially increasing effective sample availability.Stochastic augmentation can generate virtually infinite samples, although augmented examples are not independent and identically distributed.
Explicit regularization improves generalization by constraining the hypothesis class, whereas data augmentation exploits domain knowledge rather than simply reducing representational capacity.The paper contrasts this domain-specific bias with the more generic restrictions imposed by explicit methods.
Dropout can be viewed as random input-space augmentation without domain knowledge, making its generalization bounds pessimistic relative to domain-specific augmentation.The paper presents a similar comparison for weight decay.

4 Methods

The experiments compare three network architectures across ImageNet, CIFAR-10, and CIFAR-100 under different combinations of explicit regularization and data augmentation. Training settings largely follow the original architecture papers, enabling systematic comparisons.

Light augmentation uses horizontal flips and 10% horizontal and vertical translations, while heavier augmentation adds affine transformations, contrast, brightness adjustment, and ImageNet random crops.The heavier scheme was intentionally not designed as a particularly successful augmentation method.
The study evaluates All-CNN, WRN, and DenseNet on ImageNet, CIFAR-10, and CIFAR-100.These are popular architectures and benchmark datasets for visual object recognition.
The training hyperparameters, including learning rate, epochs, batch size, and optimizer, are kept as in the original papers.This preserves the published training configurations while varying regularization and augmentation conditions.
Each architecture is trained with and without weight decay and dropout, and with no, light, or heavier data augmentation.Performance is measured on held-out test sets.

5 Results

Across benchmark experiments, data augmentation alone matched or exceeded explicit regularization, while remaining more adaptable to reduced data and architectural changes. Explicit regularization also slowed learning dynamics and could reduce performance when conditions changed.

5.1 An alternative to explicit regularization: Data augmentation alone improved accuracy by 8.57%, versus 7.90% when combined with explicit regularization.The comparison covered 16 cases, with augmentation alone outperforming the combined approach in 9/16 cases; the bootstrap difference was positive with 95% confidence (P value = 0.0220).
5.1 An alternative to explicit regularization: Removing weight decay and dropout improved then state-of-the-art accuracy in 4 of 8 studied cases while keeping other hyperparameters unchanged.The original models included light augmentation, weight decay, and dropout, with parameters presumably tuned for accuracy.
5.1 An alternative to explicit regularization: Weight decay and dropout added 3.02% average improvement over baseline, compared with 8.46% from light data augmentation.Heavier augmentation further improved CIFAR-10 and CIFAR-100 over light augmentation, but not ImageNet.
5.1 An alternative to explicit regularization: Models using only data augmentation reached high validation performance after a few epochs and depended less on learning-rate schedules than regularized models.Regularized models initially fit the training data less effectively and relied heavily on learning-rate decay for their final performance.
5.2 Fewer available training examples: With 50% of the data, explicit regularization without augmentation reached 83.20% of original accuracy, below 88.11% without explicit regularization.At 10% of the data, the corresponding averages were 58.75% and 58.72%, respectively.
5.2 Fewer available training examples: With augmentation and reduced data, models without explicit regularization averaged 91.64% versus 88.78% at 50% data, and 68.12% versus 61.16% at 10%.The performance gap increased as training data decreased, with bootstrap mean differences of 2.78 and 6.96 percentage points.

6 Discussion

The discussion argues that data augmentation can replace much of the role of weight decay and dropout, while offering better adaptability and lower tuning costs. It also frames this conclusion as practically useful but empirically bounded by the study’s design and scope.

The authors challenge the routine inclusion of weight decay and dropout, proposing data augmentation as a first-class training technique.
The empirical conclusions are limited by the number of experiments, fixed original hyperparameters, unoptimized heavier augmentation, and domain-specific expert knowledge requirements.The study trained over 300 models across three architectures and three datasets, but retained hyperparameters from original models and did not optimize the heavier augmentation scheme.
Across most experiments, data augmentation alone matched or exceeded models using both augmentation and explicit regularization, while differences in remaining cases were small.
Weight decay and dropout add computation and usually require repeated training with sensitive hyperparameter choices that may not transfer across architectures or training-data amounts.
6.1 Do deep nets really need weight decay and dropout?: All-CNN on CIFAR-10 reached about 93.3 % accuracy with light augmentation alone, and remained stable when layers were added or removed.With three more layers it reached 93.4 %; with three fewer layers it reached about 90 %, unlike explicitly regularized models whose performance dropped more sharply.
6.2 Rethinking Data Augmentation: Data augmentation can preserve representational capacity, improve robustness through plausible transformations, and adapt across architecture depths and training-data amounts without specific hyperparameter fine-tuning.

A Details of network architectures

The appendix describes three architectures used in the main experiments: All-CNN, Wide Residual Network, and DenseNet, which differ in depth, connectivity, and parameter efficiency.

The main experiments use All-CNN, Wide Residual Network, and DenseNet as distinct network architectures.
All-CNN is relatively small and simple, WRN is deeper with residual connections and more parameters, and DenseNet is much deeper with dense connections and greater parameter efficiency.

A.1 All Convolutional Network

The appendix specifies the All-CNN, WRN, and DenseNet architectures and their training configurations used in the experiments. It also records architecture-specific dimensions, parameter counts, and optimization schedules.

All-CNN: All-CNN uses only convolutional layers with ReLU activation and has 16 layers with 9.4 million parameters for ImageNet and 12 layers with about 1.3 million parameters for CIFAR.
All-CNN: The experiments also include shallower and deeper All-CNN variants with 9 layers and 374,000 parameters, or 15 layers and 2.4 million parameters.
Training configuration: Training follows the original papers’ settings, including stochastic gradient descent, fixed Nesterov momentum, scheduled learning-rate decay, specified batch sizes, and He initialization.
Wide Residual Network: WRN-28-10 has 28 layers and about 36.5 million parameters, using wider layers than ResNet to achieve better performance with fewer layers.
DenseNet: DenseNet-BC uses densely connected blocks with bottleneck compression rate θ = 0.5, growth rate k = 12, and nearly 0.8 million parameters.

B Details of the heavier data augmentation scheme

The heavier augmentation scheme expands training images through affine, contrast, and brightness transformations. These operations alter image geometry or appearance while producing transformed examples for training.

Affine transformations: The heavier augmentation scheme includes affine transformations applied to the input images.
Appearance transformations: Contrast adjustment rescales deviations from the image mean using γ, while brightness adjustment adds an offset δ to the image.

C Detailed and extended experimental results

Extended experiments compare explicit regularization, batch normalization, and augmentation across architectures, data quantities, and architecture variants. Data augmentation is generally more consistent and robust than explicit regularization, while explicit regularization can degrade performance when settings are mismatched.

Augmentation and regularization: Light augmentation substantially outperforms training on the original data alone, and heavier augmentation almost always further improves test accuracy without explicit regularization.
Augmentation and regularization: The interaction of weight decay and dropout is inconsistent: some cases favor both, whereas others favor dropout alone.
Batch normalization: Batch normalization combines well with data augmentation but has inconsistent effects with explicit regularization, including a drastic reduction with dropout alone on unaugmented CIFAR-10 and CIFAR-100.
Reduced-data experiments: With 80%, 50%, 10%, and 1% of the available data, augmentation alone better resists reduced training data than explicit regularizers, although 1% results show larger variability.
Architecture adaptability: Both shallower and deeper All-CNN variants perform much worse with explicit regularization, indicating that its hyperparameters require careful tuning for architectural changes.

D Norm of the weight matrix

The appendix examines Frobenius norms of learned weight matrices as a rough complexity estimate. Heavier augmentation tends to produce larger norms, but weight norms alone do not explain the large performance differences between models.

Norm analysis: Heavier data augmentation seems to yield solutions with larger Frobenius norms of the learned weight matrices.
Interpretation: The poor performance of regularized shallower and deeper All-CNN models is not fully explained by the norms of their learned weights.
Interpretation: The exact reasons why some regularized models fail to generalize well require more thorough analysis and are left for future work.

E Carbon footprint of the computational experiments

The paper treats the environmental cost of deep-learning experiments as significant and reports estimated energy use and carbon emissions for its computational setup. Its complete experiments required 3,276 GPU hours and produced an estimated 390.45 kg CO2e.

Training large neural networks consumes considerable energy, while model compute has increased exponentially over the last decade.
The study reports estimated carbon emissions for each architecture and dataset based on hardware characteristics and an online carbon-emissions calculator.
The paper includes tables summarizing the computational setup and estimated carbon emissions associated with training the models.
3,276 GPU hours corresponded to 832.53 kWh and an estimated 390.45 carbon dioxide equivalent for the complete experiments.The reported GPU time equals 136.5 days of actual real time because the authors used a single GPU.

F On the taxonomy of regularization

The paper introduces explicit and implicit regularization as a complete distinction for describing regularization effects in deep networks. It also separates perceptually plausible, domain-specific augmentation from other data-dependent manipulations.

The paper contributes definitions of explicit and implicit regularization because these concepts have been used ambiguously in prior literature.
Earlier reviews discussed regularization methods and implicit effects such as SGD without developing the explicit–implicit distinction.
The paper argues that data-independent versus data-dependent regularization leaves ambiguity for techniques such as batch normalization.
Implicit regularization covers any regularization effect that does not come from explicit, or data-independent, techniques.
Domain-specific data augmentation should create perceptually plausible samples from the same underlying probability distribution as the available data.