Source-linked AI summary
Regularization for Deep Learning: A Taxonomy
Jan Kukačka, Vladimir Golkov, Daniel Cremers
TL;DR
Regularization has multiple definitions, while existing methods are often studied separately. This paper proposes a systematic taxonomy spanning five training elements, demonstrates links among methods, and offers recommendations for choosing and developing regularization techniques.
Problem
Regularization has multiple definitions, while regularization methods are often studied separately from one another.
Method
The paper proposes a systematic taxonomy covering data, network architecture, error term, regularization term, and optimization procedure.
Results
The taxonomy describes methods by category and subcategory, demonstrates links among existing methods, and supports discovery of new methods by combining their properties.
Takeaways & Limitations
The taxonomy helps users choose combinations of regularization techniques and helps developers identify principles and promising combinations for new methods.
Takeaways & Limitations
The taxonomy acknowledges ambiguity in assigning methods to categories and notes uncertainty about whether some optimization methods improve generalization.
Abstract
from arXiv · showhide
Regularization is one of the crucial ingredients of deep learning, yet the term regularization has various definitions, and regularization methods are often studied separately from each other. In our work we present a systematic, unifying taxonomy to categorize existing methods. We distinguish methods that affect data, network architectures, error terms, regularization terms, and optimization procedures. We do not provide all details about the listed methods; instead, we present an overview of how the methods can be sorted into meaningful categories and sub-categories. This helps revealing links and fundamental similarities between them. Finally, we include practical recommendations both for users and for developers of new regularization methods.
1 Introduction
The paper broadens regularization beyond loss penalties to any supplementary technique intended to improve test-set generalization. It introduces a taxonomy to organize diverse methods and support practical use.
- The broader definition encompasses modifications to learning algorithms intended to reduce test error without reducing training error.
- Regularization is defined broadly as any supplementary technique that aims to make a model generalize better on the test set.
- The taxonomy organizes regularization methods into meaningful categories and subcategories rather than fully summarizing every referenced method.
- The paper includes practical recommendations for users and developers of new regularization methods.
2 Theoretical framework
The theoretical framework describes neural-network training as minimizing empirical risk over trainable weights. It identifies five elements that determine learned weights and therefore provide top-level categories for regularization.
- A neural network is a trainable function f_w mapping inputs x to outputs y, with training seeking weights w that minimize a loss function.
- The loss separates into an error function E, which evaluates predictions against targets, and a regularization term R, which penalizes models using other criteria.
- Because the data distribution P is unknown, expected-risk minimization is approximated using empirical risk on a sampled training set D.
- The taxonomy treats the training set D, model family f, error function E, regularization term R, and optimization procedure as potential regularization elements.
- The paper acknowledges ambiguity in assigning methods to categories and subcategories.
3 Regularization via data
Data-based regularization transforms training data through deterministic or stochastic procedures, including representation changes, augmentation, and domain-specific transformations. Target-preserving augmentation replaces original samples with transformed samples and can address limited-data or class-imbalance settings.
- Regularization via data: Data-based regularization transforms a training set D into a new set D_R through preprocessing, feature extraction, or sample generation.
- Regularization via data: A stochastic transformation τ_θ uses parameters drawn from a probability distribution, enabling generation of additional samples.
- Regularization via data: Data-based methods are categorized by transformation properties, parameter determinism or stochasticity, sampling strategy, representation, application location, domain specificity, and parameter dependence.
- Regularization via data: Adaptive sampling selects transformation parameters through optimization, often seeking challenging samples with high network error or prescribed fake-target behavior.
- Target-preserving data augmentation: Target-preserving augmentation replaces (x_i,t_i) with (τ_θ(x_i),t_i) sampled from Q, attempting to bridge limited-data differences between expected and empirical risk.
- Target-preserving data augmentation: For class imbalance, undersampling or oversampling may deliberately make Q less realistic while producing better models.
- Summary of data-based methods: The paper groups target-preserving augmentation, Dropout, and Batch normalization as methodologically close data-based techniques, while listing generic and domain-specific methods separately.
4 Regularization via the network architecture
Architecture-based regularization selects model families whose structure encodes assumptions about the desired input-output mapping. The section also highlights overlaps among architectural, stochastic, and data-based methods.
- Assumptions about the mapping: Choosing an architecture can regularize by hardwiring assumptions about the input-output mapping and limiting the search space of models.These assumptions may be imposed hard or soft, helping select mappings that are neither too simple nor too complex.
- Architecture design: Architectural assumptions can be encoded in layer operations or in the connections between layers.Table 3 distinguishes these forms and notes partial overlaps among listed methods.
- Weight sharing: Weight sharing reduces model complexity and can encode prior knowledge such as locality and shift-equivariance.Convolutional networks provide the main example discussed.
- Activation functions: Activation-function choices affect training times and accuracy, while some activations are designed explicitly to provide regularization effects.ReLUs help avoid vanishing gradients and provide expressive mapping families; Maxout and stochastic pooling are cited as regularizing examples.
- Overlapping categories: Noise-based models such as Dropout inject stochasticity, and hidden-feature transformations can be classified under both data and architecture.The taxonomy therefore permits methods to occupy more than one branch.
5 Regularization via the error function
The error function can itself regularize when its notion of prediction quality incorporates assumptions about the data or robustness to particular conditions.
- 5 Regularization via the error function: Error functions can have a regularizing effect in addition to measuring consistency between predictions and targets.Dice coefficient optimization is given as an example because it is robust to class imbalance.
6 Regularization via the regularization term
Regularization terms add target-independent penalties that encode desired model properties and inductive biases. The taxonomy organizes these terms by their dependencies and connects several apparently different methods.
- Role of the regularization term: A regularization term adds a target-independent penalty that encodes properties of the desired model and can be evaluated on unlabeled samples.Unlike the error function, it need not measure consistency with targets.
- Weight-based terms: Weight decay penalizes the weights directly and corresponds from a Bayesian perspective to a symmetric multivariate normal prior.The paper identifies weight decay as the most popular regularization term and reports successful use.
- Smoothness: Smoothness regularization encourages nearby inputs to produce nearby outputs by penalizing mappings with large input derivatives.The Jacobian penalty is used in contractive autoencoders.
- Taxonomy of dependencies: Loss regularizers are categorized by dependence on weights, outputs, derivatives with respect to weights or inputs, and derivatives of the error.The Jacobian penalty depends on w, f, and x, whereas weight decay depends only on w.
- Cross-branch relations: Several methods across taxonomy branches are equivalent or approximately equivalent, including noise injection and Jacobian penalties, and fast dropout and deterministic approximations.These relationships are recorded in the paper’s equivalence mappings.
7 Regularization via optimization
Optimization procedures regularize through initialization, updates, and termination, with SGD receiving particular attention. Short training can act as implicit regularization, but the generalization role of optimization methods is not always clear.
- Taxonomy of optimization methods: The optimization branch is divided into initialization or warm-start methods, update methods, and termination methods.SGD and its derivatives are the main focus within this branch.
- Update methods: SGD updates weights using the gradient of the loss on a mini-batch, with mini-batch variation adding noise that can help escape saddle points.Supplementary gradient noise can reinforce this effect.
- Implicit regularization: Under certain mild assumptions, reaching low training error in reasonable time allows SGD to generalize well and act as an implicit regularizer.The paper also cites experiments finding that Dropout, data augmentation, and weight decay are neither necessary nor sufficient for good generalization.
- Initialization and warm-start methods: Initialization strategies tune weight distributions to maintain activation variance and prevent vanishing or exploding activations and gradients.Pre-training and curriculum learning provide complementary warm-start strategies.
- Caveat: It remains unclear which optimization methods merely accelerate optimization and which improve generalization; some methods can lose SGD’s regularization abilities.AdaGrad and Adam are cited as examples in this caution.
- Update methods: Dropout corresponds to optimization steps performed in subspaces of weight space after selected neuron weights are zeroed and before discarded coordinates are restored.The figure depicts this as successive projection, descent, and restoration steps.
- Termination methods: Termination methods include early stopping with a validation set and fixed iteration counts when a validation set is unavailable.Stopping early can reduce fitting to training-sample noise.
8 Recommendations, discussion, conclusions
The paper presents a taxonomy-based view of regularization and recommends choosing techniques according to available data, prior knowledge, and the intended input-output mapping. It also identifies data augmentation and combinations of existing properties as promising directions for users and method developers.
- Recommendations for users: Users should exploit information in the data and prior knowledge, begin with popular methods, and tune architectures, optimizers, and learning rates pragmatically.The recommendations include considering good random initialization, trying several optimizers, and selecting parameters such as learning rate carefully.
- Recommendations for users: When data are limited, gathering additional real data is advisable, while labeled, same-domain, reliable samples are preferred but alternatives can still help.Unlabeled, similar-domain, lower-quality, or additional-task data may remain useful when appropriately handled.
- Recommendations for users: Known meaningful transformations can be encoded through target-preserving augmentation, while unknown or insufficient transformations may be inferred from data.Popular generic methods such as advanced Dropout variants may also help.
- Architecture and regularization terms: Architectures and regularization terms can encode prior knowledge about mappings, but hardwiring assumptions is harsher than the tunable influence of data and loss terms.Convolutional layers are appropriate when local and shift-equivariant feature extraction on regular-grid data is desired.
- Recommendations for developers: Data augmentation is more expressive than loss terms because it can use rich transformation distributions rather than only infinitesimal neighborhoods around training samples.The authors suggest that understanding successful methods can reveal promising combinations and empty niches for new research.
- Future directions: Adaptive sampling of transformation parameters, class-dependent transformations, and stronger regularization for adversarial robustness are identified as directions needing further investigation.Adaptive sampling might lead to lower errors and shorter training times, while class-dependent transformations might produce more plausible samples.
- Conclusions: The taxonomy organizes regularization around data, architecture, error terms, regularization terms, and optimization procedures.It provides an overview for choosing combinations of techniques and identifying principles for new methods.
Acknowledegments
The authors acknowledge Antonij Golkov for valuable discussions and report ERC Consolidator Grant “3DReloaded” support.
- The paper acknowledges Antonij Golkov for valuable discussions and ERC Consolidator Grant “3DReloaded” support.
A Ambiguities in the taxonomy
The taxonomy contains unavoidable ambiguities because methods can be represented in multiple equivalent ways across data, architecture, weights, and error-term categories. The authors resolve these cases using common notions and Occam’s razor.
- Input transformations and architectures: Splitting the mapping y = f_w(τ_θ(x)) between input transformation and network function is ambiguous and depends on the chosen perspective.The same method may therefore fit Section 3 or Section 4 depending on how the mapping is decomposed.
- Stochastic methods: Stochastic depth can be interpreted as stochastic architecture, weight, or hidden-feature transformation depending on where the randomness is assigned.The paper illustrates alternatives involving dropped connections, zeroed weights, or dropout masks applied to hidden features.
- Dropout representations: Dropout noise can be represented through a mask, a pseudorandom seed, or an input transformation that internally generates the mask.The authors prefer the formulation whose parameters directly describe the input transformation without extra generalizations.
- Layers: Network functions can be decomposed into layers in several ways, and transformations such as Dropout or Batch normalization may also be called layers.This creates overlap between input or hidden-feature transformations and architecture-based regularization.
- Weight sharing: Weight sharing is ambiguous when equivalent formulas place the same trainable parameter once in one formulation and multiple times in another.The classification can depend on the chosen mathematical representation.
- Error and regularization terms: Auxiliary denoising losses can be categorized either as regularization terms without auxiliary targets or as error terms using ideal reconstructions as targets.The distinction depends on how targets are defined in the formulation.
B Data-augmented loss function
The data-augmented loss replaces training samples with transformed samples and can be interpreted through the induced augmented-data distribution. Its relationship to expected risk is governed by importance-sampling reweighting.
- Augmented loss: Target-preserving augmentation replaces each training input x_i with τ_θ(x_i) while retaining its target t_i in the empirical-risk loss.The transformation parameters generate an augmented distribution Q used for training.
- Augmented distribution: The induced density q(x,t) defines a valid probability distribution Q when it is non-negative and integrates to one.The construction uses the transformation process to characterize the distribution of augmented data.
- Expected risk: When Q = P, the augmented-data loss becomes the expected risk.Here P is the ground-truth data distribution and Q is the distribution induced by augmentation.
- Importance sampling: The difference between expected loss L and augmented-data loss L̂_A is the importance-sampling reweighting term p(x,t)/q(x,t).As Q more closely models P, the augmented-data loss more closely approximates expected loss and bridges empirical and expected risk.