Source-linked AI summary
The Effects of Regularization and Data Augmentation are Class Dependent
Randall Balestriero, Leon Bottou, Yann LeCun
TL;DR
Regularization is used to reduce overfitting, and this paper examines whether DA and weight decay affect classes uniformly. Across the studied settings, regularization can improve average performance while reducing accuracy for particular classes, including after transfer.
Problem
Method
The study combines an explanation of how DA can induce bias with class-level analyses of DA and weight decay, plus transfer evaluation on INaturalist.
Source: §2 · p. 3, §2.2 · p. 5, §2.3 · p. 6Results
Regularization improves average test performance while causing significant drops on some classes, and pretraining regularization also changes per-class downstream performance.
Source: §3 · p. 8, §2.5 · p. 8Takeaways & Limitations
Aggregate performance alone can miss potentially harmful class-specific losses, especially when models are used for transfer learning.
Source: §3 · p. 8The study focuses on class-level analysis and defers the finer-grained sample-level analysis to future work.
Source: §2.3 · p. 7
Abstract
from arXiv · showhide
Regularization is a fundamental technique to prevent over-fitting and to improve generalization performances by constraining a model's complexity. Current Deep Networks heavily rely on regularizers such as Data-Augmentation (DA) or weight-decay, and employ structural risk minimization, i.e. cross-validation, to select the optimal regularization hyper-parameters. In this study, we demonstrate that techniques such as DA or weight decay produce a model with a reduced complexity that is unfair across classes. The optimal amount of DA or weight decay found from cross-validation leads to disastrous model performances on some classes e.g. on Imagenet with a resnet50, the "barn spider" classification test accuracy falls from $68\%$ to $46\%$ only by introducing random crop DA during training. Even more surprising, such performance drop also appears when introducing uninformative regularization techniques such as weight decay. Those results demonstrate that our search for ever increasing generalization performance -- averaged over all classes and samples -- has left us with models and regularizers that silently sacrifice performances on some classes. This scenario can become dangerous when deploying a model on downstream tasks e.g. an Imagenet pre-trained resnet50 deployed on INaturalist sees its performances fall from $70\%$ to $30\%$ on class \#8889 when introducing random crop DA during the Imagenet pre-training phase. Those results demonstrate that designing novel regularizers without class-dependent bias remains an open research question.
1 Introduction
The introduction frames regularization as a way to reduce overfitting, then argues that DA and weight decay can improve aggregate accuracy while unevenly harming classes.
1 Introduction
- Regularization reduces overfitting by narrowing the performance gap between training and test samples.The introduction notes that this is useful when training data are finite and the system’s optimal design is unknown.Source: §1 · p. 1
- DA adds artificially generated training samples, whereas weight decay is an explicit parameter-based regularizer.The introduction notes that DA requires more domain knowledge to succeed than weight decay.Source: §1 · pp. 1–2
- Structural risk minimization selects model complexity by balancing underfitting and overfitting, but regularization can leave class-conditional risks misaligned.The figure contrasts class-specific optima with an ideal setting in which the same complexity is optimal for all classes.Source: Figure 1 · p. 2
- Regularization can improve average test accuracy while making some classes arbitrarily inaccurate.The paper identifies this class favoritism for both DA and weight decay.Source: §1 · p. 2
- Regularization’s bias can reduce estimator variance, but class-differential bias is potentially dangerous, including in transfer learning.The introduction distinguishes the usual bias-variance trade-off from bias that treats classes differently.Source: §1 · p. 2
2 Regularization Creates Class-Dependent Model Bias that can be Harmful even for Transfer Learning Tasks
The paper argues that regularization can improve average test performance while introducing class-dependent bias, and studies how this bias arises, varies across classes, and transfers to downstream tasks. Across data augmentation and weight decay, the authors find that class-level performance can decline even as average performance improves, including after transfer from ImageNet to INaturalist.
2.1 When Data-Augmentation Creates Bias
- When a transformation moves samples across level sets of the true function, data augmentation can induce irreducible bias in a sufficiently capable model that minimizes training loss.The paper calls augmentation label-preserving when its transformations remain on the same level set; otherwise, bias follows unless targets are adjusted for the shift.Source: §2.1 · pp. 3–4
2.2 The Same Data-Augmentation can be Label-Preserving or Not Between Different Classes
- The same augmentation can preserve label information for some classes but remove it for others, creating an imbalance in informative augmented samples across classes.Per-class test-accuracy trends were consistent across the evaluated architectures; the authors say the resulting models focus on classes for which the augmentation is most label-preserving.Source: §2.2 · p. 5, Figure 3 · p. 5
2.3 Measuring the Average Treatment Effect of Data-Augmentation on Models’ Class-Dependent Bias
- Stronger random-crop augmentation raises average test accuracy while lowering some per-class accuracies; academic gown accuracy falls from 62% to 40% as τ decreases.The sensitivity analysis varies the random-crop lower bound τ on ImageNet and averages metrics over 20 trained models.Source: §2.3 · p. 6, Figure 4 · p. 6
- A one-sided Welch’s t-test found evidence that augmentation did not increase per-class test accuracy for 4.4% of ImageNet classes at 95% confidence and 2.1% at 99% confidence.The analysis is per-class; the authors leave finer-grained per-sample treatment-effect analysis for future work.Source: §2.3 · pp. 6–7
2.4 Uninformed Weight Decay Also Creates Class-Dependent Model Bias
- Varying weight decay produces different test-accuracy sensitivities across classes, with some improving and others declining.The reported ResNet50 ImageNet comparison varies the weight-decay coefficient γ while using horizontal flip as the only augmentation.Source: §2.4 · p. 7, Figure 5 · p. 7
2.5 The Class-Dependent Bias Transfers to Other Downstream Tasks
- Selecting an ImageNet-pretrained model by average accuracy can leave it most biased against classes of interest on INaturalist.The study transfers a ResNet50 with frozen backbone weights and a tuned linear classifier; crop settings vary during ImageNet pretraining.Source: §2.5 · pp. 7–8
3 Conclusion
Regularization raises average test performance while significantly lowering performance on some specific classes. Varying regularization during pre-training also changes per-class performance on downstream tasks.
3 Conclusion
- Regularization increases average test performance at the cost of significant drops on some specific classes.Source: §3 · p. 8
- Models selected for aggregate performance can be potentially harmful when deployed in transfer-learning tasks.Source: §3 · p. 8
- Changing pre-training regularization affects per-class performance on downstream tasks, including transfer from Imagenet to INaturalist.Source: §3 · p. 8
A.1 Data-Augmentation, Regularization and Structural Risk Minimization
Regularization trades variance against bias, and structural risk minimization selects among models of differing complexity using validation performance. Data augmentation can act as regularization, but may bias a model when its augmented data are misaligned with the true distribution.
A.1 Data-Augmentation, Regularization and Structural Risk Minimization
- Aggressive regularization or low model complexity reduces variance while increasing bias; weaker regularization in overparameterized models reverses that tradeoff.Source: §A.1 · p. 13
- Structural risk minimization searches nested model spaces and selects the model with the best validation performance; cross-validating weight decay is one example.Source: §A.1 · p. 13
- Tikhonov regularization can be represented as a constraint on parameter norm, and gradually reducing its strength yields nested functional spaces.Source: §A.1 · p. 14
- Data augmentation can be treated as regularization, but misalignment between augmented and true data distributions can increase model bias.Source: §A.1 · p. 14
- Prior work proposes adapting augmentation policies to measured dataset bias, and reports that augmentation can be designed to reduce that bias.Source: §A.1 · p. 14
A.2 Known Tradeoff Between Regularization, Generalization and Robustness
Prior work reports that data augmentation can worsen estimation error even with label-consistent transformations, and studies connect augmentation with an accuracy–robustness tradeoff. Other work examines bias in learned augmentation generated by biased GANs.
A.2 Known Tradeoff Between Regularization, Generalization and Robustness
- The minimum-norm interpolant trained on original plus augmented data can have larger standard error than the original-data interpolant, even with label-consistent augmentation.Source: §A.2 · pp. 14–15
- Even perfectly adapted augmentations face an inherent accuracy–robustness tradeoff that persists in the infinite-data limit.Source: §A.2 · p. 15
- Prior studies of learned augmentation note that GAN-generated data inherit biases present in the GAN.Source: §A.2 · p. 15
B Proof of theorem 1
The proof groups samples linked by transformations into equivalence classes and decomposes the empirical error over those groups. Under the stated setup, data augmentation misaligned with the true function’s level sets yields positive approximation error.
B Proof of theorem 1
- The proof treats transformations as a group action and defines two images as equivalent when one can be transformed into the other.The relation is reflexive, symmetric, and transitive.Source: B Proof of theorem 1 · p. 15
- The mapping g(X) decomposes samples into generators and DA parameters whose transformations recover the dataset.These generator-parameter pairs correspond bijectively to the equivalence classes X/∼.Source: B Proof of theorem 1 · p. 15
- Per-class accuracy curves under increasing crop proportion can differ in sensitivity, while their sample-weighted average recovers Figure 2.Some classes are more likely than others to remain label-preserved after cropping.Source: Figure 7 · p. 16
- Approximation error is positive when augmentation does not respect the true function’s level sets, assuming zero training error.The derivation splits samples into training and other samples and assumes p(Tα(µ)) is constant.Source: B Proof of theorem 1 · pp. 15–16
C Additional Figures
The additional figures show random validation or training examples for earlier figures and reprise prior analyses with color jittering or a different architecture. The captions state that color jittering yields the same conclusions, with class-dependent label-preserving regimes.
C Additional Figures
- Figures 8 and 9 show random validation-set samples corresponding to Figures 4 and 5, respectively.Source: Figure 8 · p. 16, Figure 9 · p. 17
- Color jittering reprises Figure 2 and reaches the same conclusions: classes have different label-preserving regimes.Figures 10 and 11 provide the reprises.Source: Figure 10 · p. 18, Figure 11 · p. 19
- Figure 12 reprises Figure 5 with DenseNet121 instead of ResNet50, using PyTorch implementations trained on ImageNet with varying weight decay.Source: Figure 12 · p. 20
- Figure 13 shows random samples from the INaturalist mini training set corresponding to Figure 6.Source: Figure 13 · p. 21