Source-linked AI summary
DisturbLabel: Regularizing CNN on the Loss Layer
Lingxi Xie, Jingdong Wang, Zhen Wei, Meng Wang, Qi Tian
TL;DR
CNN over-fitting motivates regularization, but existing approaches act on weights, inputs, hidden units, or model structure. DisturbLabel perturbs a small subset of labels during each mini-batch, regularizing the loss layer; experiments report competitive performance, complementary benefits with Dropout, and an implicit ensemble interpretation.
Problem
CNN training requires regularization to combat over-fitting, motivating alternatives to existing weight, input, and hidden-layer approaches.
Method
DisturbLabel randomly changes a small subset of ground-truth labels in each mini-batch, adding noise to the loss and back-propagated gradients.
Results
DisturbLabel consistently improves training by preventing over-fitting and achieves competitive performance across several image-classification benchmarks, with further gains when combined with Dropout.
Takeaways & Limitations
DisturbLabel provides loss-layer regularization complementary to Dropout and can be interpreted as implicitly averaging models trained on different noisy datasets.
Takeaways & Limitations
The ImageNet evaluation covers only AlexNet, while cooperation with other architectures is presented as an expectation rather than an evaluation.
Abstract
from arXiv · showhide
During a long period of time we are combating over-fitting in the CNN training process with model regularization, including weight decay, model averaging, data augmentation, etc. In this paper, we present DisturbLabel, an extremely simple algorithm which randomly replaces a part of labels as incorrect values in each iteration. Although it seems weird to intentionally generate incorrect training labels, we show that DisturbLabel prevents the network training from over-fitting by implicitly averaging over exponentially many networks which are trained with different label sets. To the best of our knowledge, DisturbLabel serves as the first work which adds noises on the loss layer. Meanwhile, DisturbLabel cooperates well with Dropout to provide complementary regularization functions. Experiments demonstrate competitive recognition results on several popular image recognition datasets.
1. Introduction
CNNs achieve strong image-recognition performance but remain vulnerable to over-fitting, motivating regularization methods. DisturbLabel regularizes CNNs at the loss layer by perturbing labels during training.
- CNNs have achieved significant performance gains in image recognition, supported by large image repositories and high-performance computing resources.
- Existing CNN regularization methods include weight decay, Dropout, DropConnect, data augmentation, and early stopping.
- DisturbLabel randomly assigns incorrect ground-truth labels to a small subset of each mini-batch, producing noisy losses and gradients.
- DisturbLabel regularizes the CNN on the loss layer and is presented as an alternative to combining models trained on different noisy datasets.
- Experiments report comparable performance with Dropout and better performance when DisturbLabel is combined with Dropout on several image-classification benchmarks.
2. Related Work
Prior work addresses CNN over-fitting through architectural, optimization, input, hidden-unit, and weight-level interventions. DisturbLabel instead introduces noise at the label and loss layer while assuming the original labels are correct.
- CNN research has explored deeper structures, nonlinear activations, pooling operations, optimization, and regularization to improve recognition models.
- Table 1 compares different CNN regularization techniques, with detailed references provided in the accompanying texts.
- Weight decay constrains parameters through ℓ2-regularization, while early stopping limits training before convergence.
- Data augmentation perturbs inputs, Dropout discards hidden-neuron responses, DropConnect selects weights, and Stochastic Pooling randomizes pooling inputs.
- Noisy-label research studies datasets whose labels may be inaccurate, whereas DisturbLabel assumes correct labeling and changes labels only with small probability per mini-batch.
- DisturbLabel differs from input-noise methods by regularizing the neural network at the loss layer, related to the output unit.
3. The DisturbLabel Algorithm
DisturbLabel modifies labels independently within each mini-batch before the usual SGD update, using a noise rate α. Experiments associate moderate noise with improved generalization, while the convex analysis identifies a damping effect distinct from ℓ2-regularization.
- The DisturbLabel Algorithm: DisturbLabel trains a CNN on dataset pairs (x_n, y_n), where inputs are D-dimensional vectors and labels are C-dimensional one-hot vectors.
- The DisturbLabel Algorithm: For each sample in a mini-batch, DisturbLabel randomly generates a disturbed label vector before updating parameters with the standard gradient rule.
- The DisturbLabel Algorithm: The noise rate α controls label disturbance: the true class receives probability 1 − (C−1)/C · α, while every other class receives α/C.
- The DisturbLabel Algorithm: At α = 0%, DisturbLabel reduces to ordinary training; as α approaches 100%, labels are largely discarded and training becomes nearly unsupervised.
- The DisturbLabel Algorithm: LeNet trained with α = 90% achieves < 2% testing error on MNIST, showing that relatively large noise can still produce an efficient network.
- The DisturbLabel Algorithm: Moderate noise rates such as 10% or 20% yield higher recognition accuracy than unregularized training on MNIST and CIFAR10.
- DisturbLabel as a Regularizer: Without regularization, training error falls rapidly while testing error remains high; DisturbLabel slows training-error reduction but lowers testing error.
- Theoretical Analysis: In the convex linear-regression analysis, label disturbance adds a gradient term with a damping effect that differs from ℓ2-regularization.
4. Discussions
DisturbLabel is contrasted with soft labeling and interpreted as implicit model averaging, data augmentation, and complementary regularization with Dropout. Experiments examine these interpretations on MNIST and CIFAR10, including sparse-data training.
- 4.1. Difference from Soft Labeling: Soft labels produce nearly the same accuracy as ordinary training, whereas DisturbLabel significantly improves recognition accuracy on MNIST and CIFAR10.Although their expected gradients are equal, the paper reports that soft labels lack DisturbLabel’s regularization ability.
- 4.2. Interpretation as Model Ensemble: DisturbLabel can be viewed as efficiently combining exponentially many networks with the same architecture trained on different noisy datasets.Each iteration resembles training on a different noisy dataset, with massive weight sharing across the resulting models.
- 4.3. Interpretation as Data Augmentation: DisturbLabel can reinterpret disturbed-label training examples as augmented input examples paired with their original labels.The paper derives this view by projecting a noisy output back into the input space.
- 4.4. Relationship to Other CNN Training Methods: DisturbLabel is presented as distinct from noisy-label learning and applicable to other network structures, because it assumes correct ground-truth labels and adds noise during training.The paper places its regularization on the loss layer rather than the input, hidden units, connections, or pooling operations.
5. Cooperation with Dropout
DisturbLabel and Dropout regularize different parts of CNNs and can be combined as complementary methods. Their combination improves recognition when the regularization strength is adjusted to avoid under-fitting.
- Complementary regularization: DisturbLabel regularizes the loss layer, whereas Dropout regularizes hidden layers and changes network structures during training.DisturbLabel approximates an ensemble trained on different noisy datasets; Dropout approximates an ensemble with different structures trained on the same data.
- Empirical cooperation: Combining DisturbLabel with Dropout generally improves recognition accuracy on MNIST and CIFAR10.The experiments vary DisturbLabel’s noise level while holding Dropout’s drop rate fixed.
- Noise-level effects: On MNIST, α = 10% is best with Dropout, whereas α = 20% performs much worse.Without Dropout, α = 10% and α = 20% produce comparable results.
- Regularization strength: Because both methods add regularization, their strengths must be reduced together to prevent under-fitting.Later experiments halve α when both methods are used; for strong ImageNet label noise, the Dropout rate is also decreased.
6. Experiments
DisturbLabel is evaluated across digit, natural-image, and large-scale visual-recognition datasets using LeNet, BigNet, and AlexNet configurations. The experiments report consistent or competitive gains, including improved results with Dropout and data augmentation, while ImageNet shows smaller but measurable error reductions.
- Experimental scope: The evaluation covers MNIST, SVHN, CIFAR10, CIFAR100, and ImageNet across digit, natural-image, and large-scale recognition tasks.The study uses multiple network configurations, including LeNet, BigNet, and AlexNet.
- MNIST and SVHN: On MNIST and SVHN, DisturbLabel improves recognition accuracy with or without Dropout, reaching a 2.02% error rate on SVHN with both methods and data augmentation.The MNIST and SVHN results are summarized in Table 2.
- CIFAR datasets: On CIFAR10 and CIFAR100, DisturbLabel provides consistent accuracy gains with or without Dropout; CIFAR10 reaches a 6.98% error rate with BigNet and Dropout.CIFAR10/CIFAR100 results are summarized in Table 3, with data augmentation using random cropping and flipping.
- ImageNet: The ImageNet evaluation uses AlexNet with Dropout, lowering the Dropout rate to 0.43 because uniformly distributed wrong labels introduce strong noise.The authors evaluate DisturbLabel only on AlexNet and suggest cooperation with other architectures as a belief rather than an evaluated result.
- ImageNet: On ImageNet, AlexNet error rates decrease from 43.1% to 42.8% for top-1 and from 19.9% to 19.7% for top-5 with DisturbLabel.After about 20 epochs, the DisturbLabel model has higher recognition accuracy at each testing phase.
7. Conclusions
The paper presents DisturbLabel as a simple loss-layer regularizer that prevents over-fitting and can be interpreted as implicit model ensembling and data augmentation. Experiments report competitive image-classification performance and complementary cooperation with Dropout.
- Conclusion: DisturbLabel randomly assigns incorrect labels to a small subset of training data during each iteration to regularize CNNs on the loss layer.The paper describes this as an alternative solution for implicit model ensemble and data augmentation.
- Conclusion: DisturbLabel consistently improves network training by preventing over-fitting and cooperates well with Dropout.The conclusion characterizes the two methods as regularizing different aspects of CNNs.
- Conclusion: Experiments verify competitive performance on several popular image-classification benchmarks.