PaperlayerView exact revision on arXiv

Source-linked AI summary

Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks

Claudio Filipi Gonçalves dos Santos, João Paulo Papa

arXiv:2201.03299v1cs.CVcs.LG

TL;DR

CNNs can overfit training data, motivating regularization methods that improve generalization across image-processing tasks. This survey synthesizes recent techniques, organizes them by input, internal, and label changes, and emphasizes reproducible implementations. It reports improved results for the surveyed models, while most methods add little training cost and none changes inference time.

  • Problem

    CNNs may fit training data without generalizing well to new information, making regularization important for image-processing models.

  • Method

    The survey analyzes recent CNN regularizers, classifying them by data augmentation, internal changes, or label transformations, with emphasis on reproducibility.

  • Results

    The surveyed regularizers improved original models; most require no relevant extra time, and none affects inference time.

  • Takeaways & Limitations

    The survey provides recent methods, evaluation information, and a proposed protocol intended to help compare future regularizers.

  • Takeaways & Limitations

    The surveyed literature lacks proper investigation of lightweight CNNs such as MobileNet-V3.

Abstract

from arXiv · show

Several image processing tasks, such as image classification and object detection, have been significantly improved using Convolutional Neural Networks (CNN). Like ResNet and EfficientNet, many architectures have achieved outstanding results in at least one dataset by the time of their creation. A critical factor in training concerns the network's regularization, which prevents the structure from overfitting. This work analyzes several regularization methods developed in the last few years, showing significant improvements for different CNN models. The works are classified into three main areas: the first one is called "data augmentation", where all the techniques focus on performing changes in the input data. The second, named "internal changes", which aims to describe procedures to modify the feature maps generated by the neural network or the kernels. The last one, called "label", concerns transforming the labels of a given input. This work presents two main differences comparing to other available surveys about regularization: (i) the first concerns the papers gathered in the manuscript, which are not older than five years, and (ii) the second distinction is about reproducibility, i.e., all works refered here have their code available in public repositories or they have been directly implemented in some framework, such as TensorFlow or Torch.

1 INTRODUCTION

CNN performance depends not only on architecture and optimization but also on regularization, which limits overfitting and can improve results without changing inference cost. This survey emphasizes recent, reproducible methods organized by where they modify CNNs.

  • Motivation: Regularization can improve final results while leaving inference computational cost unchanged.Some methods add training overhead or require pre-training epochs, but the survey states that output gains usually compensate for these costs.
  • Motivation: Regularization helps CNNs generalize beyond training data by reducing overfitting.Overfitting occurs when models fit training information well but perform poorly on new data.
  • Scope: All surveyed algorithms were selected for recent development and code availability to support reproducibility.Except for Dropout, the surveyed studies are no older than four years, and implementations are available publicly or in frameworks.
  • Scope: The survey groups regularizers into data augmentation, internal changes, and label transformations.These categories modify inputs, internal network values such as kernels or weights, or desired outputs, respectively.
  • Related work: Prior surveys emphasized older general methods or dropout-based approaches, whereas this work focuses on recent developments.The authors describe their survey as covering recent strategies, including work published as late as 2021.
  • Distribution: Approximately 44% of surveyed works modify inputs, making data augmentation the most common regularization area.Internal-structure methods form the second most common group, while label regularization is less represented.

2 CONVOLUTIONAL NEURAL NETWORKS

CNNs use stacked convolutional operations to extract increasingly complex image features and have expanded across classification, detection, reconstruction, and generative tasks. Their development includes architectures such as AlexNet, VGG, Inception, ResNet, EfficientNet, and task-specific models.

  • CNN fundamentals: CNNs use 2D convolution and stacked layers to learn image features with backpropagation.Earlier layers detect simpler features such as borders, while deeper layers identify more complex traits.
  • Architectural development: AlexNet achieved first place in the 2010 ImageNet classification challenge, exceeding the runner-up by more than 10%.The model used an eight-layer CNN with an MLP classifier.
  • Architectural development: VGG showed that stacking smaller convolution kernels can outperform shallower layers with larger kernels over the same region.Inception later introduced parallelized kernel operations within a layer.
  • Architectural development: ResNet fuses outputs from equal-dimensional layers before pooling, helping backpropagation handle vanishing and exploding gradients.This residual connection became a defining architectural feature.
  • Architecture search: Neural Architecture Search uses a trained agent to discover CNNs under specified rules, but the search can require considerable time.Later work improved the search process, and EfficientNet incorporated better-designed rules and Squeeze-and-Excitation.
  • Applications: CNNs also support object detection, image reconstruction, denoising, deblocking, super-resolution, and style transfer.Examples include R-CNN and YOLO for detection, FCNs for reconstruction, and GAN-based systems for style transfer and super-resolution.

3 REGULARIZATION BASED ON DATA AUGMENTATION

Data augmentation regularizes CNNs by altering training inputs, helping models generalize better. The surveyed approaches include basic transformations, region removal, randomized erasing, and learned augmentation policies.

  • General data augmentation: Basic transformations such as flipping and noise addition can help CNNs generalize better.These changes are applied directly to training images and are visually interpretable before training.
  • Cutout: Cutout randomly removes image regions before each input is fed to the network.The reported best patch size was 16 × 16 for CIFAR-10, while the ideal size varies by dataset.
  • Region-based methods: Cutout and RandomErasing both regularize CNNs by modifying localized regions of the input image.Their difference is whether erased areas are simply removed or may also receive added information.
  • RandomErasing: RandomErasing extends Cutout by sometimes adding randomly generated information, such as noise, to erased regions.Unlike Cutout, it does not remove image regions on every application.

3.3 AutoAugment

AutoAugment searches for augmentation policies that improve CNN accuracy, but its computational burden motivates faster alternatives with similar results.

  • 3.3 AutoAugment: AutoAugment searches transformations, probabilities, and magnitudes using a controller RNN and child CNN.The five best policies are used to train the final evaluation model.
  • 3.3 AutoAugment: AutoAugment achieved state-of-the-art results on CIFAR-10, CIFAR-100, SVHN, and ImageNet.
  • 3.3 AutoAugment: PBA reduces augmentation-search overhead from 5,000 hours for AutoAugment on CIFAR-10 to 5 hours.PBA uses changing schedule policies rather than fixed policies and reports similar results through a new algorithm.
  • 3.3 AutoAugment: The search procedure creates a computational bottleneck because it may take longer than neural-network training.Search strategies can also be sub-optimal when policies found with shallower networks do not transfer to the best global result.

3.6 Mixup

Mixup trains on mixtures of images and labels, while CutMix replaces image regions and assigns labels according to the regions contributed by each class.

  • 3.6 Mixup: Mixup combines two training images and requires an encoding label for the resulting mixture.The resulting input-output pairs expose the model to corrupted inputs and have improved results across several tasks.
  • 3.6 Mixup: CutMix replaces entire input regions and weights each class label by its contributed image area.A 30% airplane replacement in a cat image yields a label of 70% cat and 30% airplane.
  • 3.6 Mixup: CutMix produces heat maps that more accurately highlight object-defining regions than the input-mixing approach described alongside it.The passage attributes this observation to techniques such as grad-CAM.
  • 3.6 Mixup: CutBlur is the only approach identified for super-resolution, replacing a high-resolution image area with a low-resolution region from a similar area.The survey reports that CutBlur helps generalization for super-resolution and can also be applied to Gaussian-noise reconstruction.

3.9 BatchAugment

Batch Augmentation uses the mini-batch limit to duplicate augmented images, improving final results and reducing the epochs required for convergence.

  • 3.9 BatchAugment: Batch Augmentation exploits the GPU’s mini-batch-size limit rather than using the batch only once.The supplied passage truncates the precise procedure but states that augmented images are duplicated.
  • 3.9 BatchAugment: Batch Augmentation significantly improves neural-network final results.
  • 3.9 BatchAugment: Duplicating augmented images requires fewer epochs for convergence.
  • 3.9 BatchAugment: Higher test resolution may affect inference accuracy, but the supplied passage does not state the proposed resolution relationship fully.The passage notes that training at 224 × 224 normally uses the same resolution for testing before introducing a higher-resolution test-set proposal.
  • 3.9 BatchAugment: Combining regularizers can improve results, although the survey notes that interactions among regularizers are often not evaluated.The Bag-of-Tricks example combines Mixup, Label Smoothing, and Knowledge Distillation, improving MobileNet by almost 1.5% on ImageNet.

4 REGULARIZATION BASED ON INTERNAL STRUCTURE CHANGES

Internal regularizers modify network weights, kernels, feature maps, or branch operations without explicitly changing the input, using methods such as dropout and branch perturbation.

  • 4 REGULARIZATION BASED ON INTERNAL STRUCTURE CHANGES: Internal regularizers change weights or kernel values during training without explicitly changing the input.The survey includes dropout variants and tensor-operation methods such as Shake-shake.
  • 4 REGULARIZATION BASED ON INTERNAL STRUCTURE CHANGES: Dropout randomly removes neurons during training, whereas Maxdropout deactivates outputs according to normalized activation thresholds.Maxdropout can outperform Dropout on WideResNet-28-10 and improve ResNet18 results on CIFAR-10 and CIFAR-100.
  • 4 REGULARIZATION BASED ON INTERNAL STRUCTURE CHANGES: DropBlock removes contiguous regions from feature maps to address spatial correlations among CNN neurons.TargetDrop combines attention with DropBlock to remove highly discriminative channel regions and produce more consistent class-determining regions.
  • 4 REGULARIZATION BASED ON INTERNAL STRUCTURE CHANGES: AutoDrop learns drop patterns during training, addressing DropBlock’s manually designed and fixed pattern.
  • 4 REGULARIZATION BASED ON INTERNAL STRUCTURE CHANGES: Shake-shake changes branch multipliers differently during forward and backward passes in three-branch ResNets.It improves CIFAR-10 results by at least 0.15% and almost 0.6% over the best result, while its three-branch restriction limits direct comparisons.
  • 4 REGULARIZATION BASED ON INTERNAL STRUCTURE CHANGES: ShakeDrop modifies Shake-shake perturbations and applies them to ResNeXt, ResNet, Wide ResNet, and PyramidNet.The supplied passage reports that ShakeDrop outperforms the original results for each named architecture.
  • 4 REGULARIZATION BASED ON INTERNAL STRUCTURE CHANGES: Manifold Mixup applies Mixup within any internal CNN layer rather than only at the input layer.

5 LABEL REGULARIZATION

Label regularization transforms targets to reduce overfitting and overconfidence, using approaches that smooth, interpolate, invert, or partially remove label information. The section also highlights that smoothing strength and duration require careful selection.

  • Mixup averages labels according to interpolation between two images, while Manifold Mixup applies the same idea among network layers.
  • Cutblur can invert the input and label because low- and high-resolution crops remain structurally interchangeable.
  • Cutout and RandomErasing can remove corresponding portions of labels, paralleling their removal of image regions.A crop covering 25% of an image could reduce the active-class value from 1 to 0.75.
  • Label smoothing modifies one-hot label values to reduce overfitting and overconfidence during classification.It makes softmax outputs closer to smoothed target vectors.
  • TSLA uses label smoothing only until a training point, after which labels return to 0 and 1; on ResNet18 with CIFAR-100 for 200 epochs, the best performance occurred at epoch 160.
  • Choosing an appropriate smoothing factor is difficult, while SLS estimates Bayes Estimation Error to define instance-specific label boundaries.SLS was fully evaluated on MobileNet V2 and reportedly overcame traditional label smoothing in some experiments.
  • JoCoR trains two similar networks, combines cross-entropy and contrastive losses, and updates parameters using the batch’s smallest losses.

6 METHODOLOGY

The methodology directly compares the reviewed regularizers by organizing results by model and reporting dataset results from the original and related works.

  • The study compares regularizers within model-specific tables for clearer evaluation across datasets.The reported comparisons concern classification performance on commonly used datasets.

6.1 Datasets

The evaluation uses CIFAR, ImageNet, and SVHN, which differ in scale, class structure, image characteristics, and dataset splits.

  • CIFAR-10 and CIFAR-100 each contain 60,000 32 × 32 images, with 50,000 for training and 10,000 for testing or validation.CIFAR-100 has 100 classes and is harder to classify than CIFAR-10.
  • ImageNet’s ILSVRC 2012 subset contains 1,240,000 training images and 50,000 validation images across 1,000 classes.Its scale is one order of magnitude larger than the CIFAR subsets.
  • AlexNet demonstrated the practical use of CNNs in the 2010 ILSVRC and surpassed the runner-up by more than 10% in accuracy.
  • SVHN contains house-number images with digit labels and is more difficult and larger in instance count than MNIST.
  • The commonly used SVHN version segments images into individual digits scaled to 32 × 32, with 73,257 labeled training instances and 26,032 evaluation images.

6.2 Architectures

The reviewed studies commonly use ResNet-family architectures and related variants whose depth, width, residual blocks, cardinality, activation order, and dropout differ. These architectural choices shape how regularization is applied and evaluated.

  • Fair comparisons require regularization methods to use the same architecture.
  • ResNet uses residual connections that concatenate outputs from previous layers with subsequent transformations.
  • ResNet: ResNet-18 is an 18-layer network with residual connections between blocks containing convolution and batch-normalization sequences.
  • Wide Residual Network: WRN retains residual connections but uses pre-activation blocks, placing batch normalization and ReLU before convolution.WRN also varies depth and widening through the WRN-d-k notation.
  • Wide Residual Network: WRN inserts Dropout between convolutional layers after ReLU activation to perturb batch normalization and prevent overfitting.
  • ResNeXt: ResNeXt divides the main branch into multiple streams according to cardinality.
  • PyramidNet: PyramidNet gradually increases feature-map dimensionality across subsequent layers and uses zero-padded shortcuts to combine tensors of different sizes.
  • PyramidNet: PyramidNet’s residual block removes its first ReLU because it contributes comparatively little to system nonlinearity.

7 EXPERIMENTAL RESULTS

The survey compiles classification results across common CNN architectures and datasets, showing that regularizer performance varies with the architecture, training protocol, and combinations of methods. It also identifies limited coverage of lightweight architectures and complex, imbalanced data.

  • Reported results: Table 2 reports classification error across CIFAR-10, CIFAR-100, SVHN, and ImageNet for multiple CNN architectures and regularization methods.The table caption defines the reported metric as error in percent and expands the main method acronyms.
  • Reported results: 2.0 CIFAR-10 error is reported for Shake-Shake combined with Fast AutoAugment and for Shake-Shake combined with AutoAugment.These are separate method rows in the compiled results table.
  • Architecture dependence: AutoAugment outperforms PBA with ResNeXT plus Shake-Shake on CIFAR-10, but PBA performs better than AutoAugment with PyramidNet plus ShakeDrop.The comparison reverses across the two architecture and regularization settings.
  • Evaluation protocol: The survey cautions that training-protocol differences can produce apparent gains unrelated to the regularizer itself, motivating consistent transformations, weight decay, and source-code availability.The authors note that some studies may use additional transformations or regularizers that affect comparison.
  • Evaluation protocol: The survey states that combining regularizers can be necessary for architectures such as Wide Residual Network, which already includes dropout in its layers.A new regularizer should therefore be evaluated for compatibility with existing dropout.
  • Scope boundaries: The survey identifies lightweight CNNs, quantized networks, unbalanced datasets, and non-classification tasks as underexplored evaluation settings.It specifically calls for more work on smaller designs, quantization, complex data, object detection, and image reconstruction.

8 CONCLUSION

The conclusion presents a recent, reproducible survey of CNN regularization methods organized by where they operate in the pipeline. It reports broad practical utility while highlighting protocol standardization and underexplored architectures, data, and tasks as priorities.

  • Contribution: The survey presents regularization as a tool for preventing CNN overfitting and improving final results, while reviewing recent methods and their reported outcomes.Its stated aim is to summarize recent developments, how the methods work, and their main results.
  • Practical implications: The survey reports that most regularizers add little training overhead and none adds inference-time overhead, although AutoAugment can increase training time.AutoAugment's additional cost comes from searching for better augmentation policies.
  • Organization: The reviewed methods are divided into input, internal, and label regularization according to whether they act before the network, within it, or at the output layer.This classification organizes the survey's main sections.
  • Evaluation: The methodology catalogs commonly used datasets and CNN architectures to help standardize evaluation protocols for future regularization research.The survey presents this information as useful for establishing comparable evaluations.
  • Evaluation: The conclusion emphasizes reproducible comparison through reported results, evaluation-protocol recommendations, and discussion of issues affecting new regularizers.The survey frames a trustworthy protocol as useful for comparing results and providing research insight.
  • Future directions: The survey identifies simpler architectures and unbalanced datasets as insufficiently studied and encourages regularization research beyond image classification, including object detection and image reconstruction.These are stated as gaps and directions for future work.
Loading 2201.03299v1…