PaperlayerView exact revision on arXiv

Source-linked AI summary

Noisy Softmax: Improving the Generalization Ability of DCNN via Postponing the Early Softmax Saturation

Binghui Chen, Weihong Deng, Junping Du

arXiv:1708.03769v1cs.CV

TL;DR

Early softmax saturation can reduce gradient contributions and limit SGD exploration during CNN training. The paper introduces Noisy Softmax, which injects annealed noise to postpone that saturation. Experiments report improved performance and state-of-the-art or competitive results on several datasets.

  • Problem

    The paper identifies early softmax saturation as a concern because saturated examples stop contributing useful gradients, limiting SGD exploration and sometimes accompanying convergence at a bad local minimum.

    Source: §3 · p. 3

  • Method

    Noisy Softmax injects annealed noise into softmax activations to postpone early saturation and sustain gradient propagation.

    Source: §1 · p. 2

  • Results

    The authors report that Noisy Softmax improves CNN performance and achieves state-of-the-art or competitive results on several datasets.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    The authors conclude that early softmax desaturation affects back-propagation parameter updates and improves CNN generalization.

    Source: §7 · p. 8

    The analysis simplifies to binary classification, while the authors state that multi-classification has the same mechanism.

    Source: §4 · p. 3

Abstract

from arXiv · show

Over the past few years, softmax and SGD have become a commonly used component and the default training strategy in CNN frameworks, respectively. However, when optimizing CNNs with SGD, the saturation behavior behind softmax always gives us an illusion of training well and then is omitted. In this paper, we first emphasize that the early saturation behavior of softmax will impede the exploration of SGD, which sometimes is a reason for model converging at a bad local-minima, then propose Noisy Softmax to mitigating this early saturation issue by injecting annealed noise in softmax during each iteration. This operation based on noise injection aims at postponing the early saturation and further bringing continuous gradients propagation so as to significantly encourage SGD solver to be more exploratory and help to find a better local-minima. This paper empirically verifies the superiority of the early softmax desaturation, and our method indeed improves the generalization ability of CNN model by regularization. We experimentally find that this early desaturation helps optimization in many tasks, yielding state-of-the-art or competitive results on several popular benchmark datasets.

1. Introduction

Softmax saturation can prematurely stop gradient contributions from training examples, restricting SGD exploration and sometimes contributing to overfitting or convergence at a bad local minimum. Noisy Softmax injects annealed noise into softmax activations to postpone early saturation; experiments report improved performance across several datasets.

1. Introduction

  • Premature softmax saturation can stop individual examples from contributing gradients, limiting SGD exploration and sometimes contributing to overfitting or convergence at a bad local minimum.The authors argue that fewer non-saturated contributing samples impede robust learning.Source: §1 · pp. 1–2
  • Noisy Softmax postpones early saturation by injecting annealed noise into softmax activations, aiming to sustain gradient propagation and encourage SGD exploration.The authors describe reduced overfitting and improved generalization as consequences of greater exploration.Source: §1 · p. 2
  • Noisy Softmax is presented as a drop-in replacement for standard softmax that can also be combined with other performance-improving techniques.The paper mentions neural activation functions and network architectures as examples.Source: §1 · p. 2
  • The paper reports extensive experiments on MNIST, CIFAR10/100, LFW, FGLFW, and YTF, whose impressive results demonstrate Noisy Softmax’s effectiveness.Source: §1 · p. 2

2. Related Work

Related work addresses gradient flow through neuron and layer desaturation, as well as noise injection in networks, weights, gradients, and inputs. This paper instead targets early softmax saturation by injecting noise at the loss layer.

2. Related Work

  • Replacing sigmoid with piecewise activation functions and adding skip connections are presented as neuron- and layer-desaturation approaches to improving gradient propagation.The authors distinguish their focus on early softmax saturation from those approaches.Source: §2 · p. 2
  • Prior noise-injection work covers ReLU and sigmoid activations, weights, adaptive weights, and gradients, while this work injects noise into the softmax loss layer.The authors distinguish their target from DisturbLabel, which disturbs labels.Source: §2 · p. 2
  • Random cropping, flipping, rotating, and jittering are described as input-data augmentation, and the authors say their method can also be interpreted as data augmentation.Source: §2 · p. 2

3. Early Individual Saturation

The section characterizes early individual saturation as a training example's softmax output becoming close to its ground-truth target before SGD exploration is complete. Such examples contribute negligible gradients, and their growing number reduces SGD's opportunities to explore.

3. Early Individual Saturation

  • When an example's softmax output is prematurely close to its ground-truth target, its gradient contribution to back-propagation becomes negligible.The binary-classification illustration describes softmax activation as sigmoid-like.Source: §3 · p. 3
  • As saturated examples accumulate, fewer training examples contribute gradients, leaving SGD fewer chances to explore and making convergence at a local minimum and overfitting more likely.The authors say extra data may be required to recover.Source: §3 · p. 3
  • The Figure 2 caption identifies a softmax activation function, and the accompanying text specifies that the plotted x-axis is the difference between f1 and f2.Source: Figure 2 · p. 3, §3 · p. 3

4. Noisy Softmax

Noisy Softmax postpones early softmax saturation by injecting positive noise into the true-class input, with the noise annealed during training. Its noise scale combines feature and classifier magnitudes with an angle-dependent factor, while computation for other classes remains as in standard softmax.

4. Noisy Softmax

  • Noisy Softmax subtracts positive noise from the true-class softmax input to create a saturation barrier and sustain gradient propagation.The noise is represented as n = μ + σξ, with ξ sampled from a standard normal distribution, and is constrained to be positive.Source: §4 · p. 3
  • The noise starts relatively large to encourage exploration and becomes smaller later, postponing saturation rather than preventing it.The method aims to support exploration initially while allowing model convergence later.Source: §4 · p. 3
  • Its noise scale is α∥W_yi∥∥X_i∥(1 − cos θ_yi), combining feature and classifier magnitudes with an angle-dependent annealing factor.The magnitude terms make the noise comparable to f_yi, while the angle term adaptively anneals it as the angle decreases during convergence.Source: §4 · p. 3
  • With SGD, Noisy Softmax changes the true-class input while other class logits and their gradients remain the same as in standard softmax.The paper states that the multiclass setting has the same mechanism as the binary scenario.Source: §4 · pp. 3–4

5. Discussion

The discussion examines how noise scale and formulation affect softmax saturation, optimization, and generalization. It also presents Noisy Softmax as a regularizer and relates it to data augmentation, multi-task learning, and other noise-injection or desaturation methods.

5.1. The Effect of Noise Scale α

  • Noisy Softmax with a relatively appropriate α (e.g., α2 = 0.1) obtains better recognition accuracy than ordinary softmax on all datasets; at α2 = 0.1, it outperforms original softmax.When α rises to 1, large noise causes slower convergence and worse performance than baseline because it drowns out the helpful signal.Source: §5.1 · p. 4

5.2. Saturation Study

  • 28.48% testing error is reported for Noisy Softmax on CIFAR100, versus nearly 37% for negative noise, which is associated with more severe early saturation.With Noisy Softmax, average predictions rise more slowly and remain lower early in training; the authors report that non-negative noise prevents early individual saturation.Source: §5.2 · pp. 4–5, Figure 4 · p. 4

5.3. Annealed Noise Study

  • Noisy Softmax outperforms free noise and amplitude noise on CIFAR100; annealing reduces noise over time, while fixed-amplitude noise can hinder detailed learning.The table caption identifies testing error rates for different noise formulations, but the supplied passages do not provide its cell values.Source: §5.3 · p. 5, Table 1 · p. 5

5.4. Regularization Ability

  • Noisy Softmax is described as a regularizer that reduces over-fitting; on CIFAR100, α2 = 0.1 yields 28.48% testing error versus 31.77% with α = 0.With α2 = 0.1, training error drops more slowly and remains higher than baseline while testing error is lower and continues decreasing.Source: §5.4 · p. 5
  • Noisy Softmax’s noisy inputs are described as equivalent to generating new training data, an interpretation of its regularization as data augmentation.The geometric account says the generated data include boundary examples that are helpful for discriminative feature learning.Source: §5.4 · p. 6
  • 2.46% and 0.93% testing errors on MNIST’s 600- and 6,000-instance subsets improve on the respective baseline errors of 3.82% and 1.30%.Training errors quickly approach zero in both baseline cases, and the authors report particular effectiveness when training data are limited.Source: §5.4 · p. 6

5.5. Relationship to Other Methods

  • Noisy Softmax can be viewed as multi-task learning with shared task-specific parameters, using many noise-generated tasks rather than manually designed losses.The authors also distinguish it from prior noise injection in other network layers and from DisturbLabel, and locate its desaturation effect in the loss layer rather than bottom layers.Source: §5.5 · p. 6

6. Experiments and Results

Experiments evaluate Noisy Softmax on MNIST, CIFAR10/100, LFW, FGLFW, and YTF, using one model and the same CNN architecture for standard and noisy softmax. Results favor Noisy Softmax on MNIST and CIFAR, and show improved face-benchmark performance over the baseline.

6.1. Architecture Settings and Implementation

  • The experiments use PReLU CNNs trained with SGD, with object and face tasks differing in batch size, learning-rate schedules, and test-time evaluation.Object datasets use original softmax for classification; face datasets use cosine distance after PCA reduction.Source: §6.1 · p. 7

6.3. Evaluation on CIFAR Datasets

  • Nearly 1% on CIFAR10 and more than 3% on CIFAR100: Noisy Softmax (α2 = 0.1) improves accuracy over baseline and outperforms the other methods.The comparison uses randomly cropped 30×30 images for data augmentation.Source: §6.3 · p. 7, Table 4 · p. 8

6.2. Evaluation on MNIST Dataset

  • Noisy Softmax (α2 = 0.1) outperforms standard softmax with the same architecture and is competitive with state-of-the-art methods on MNIST.Accuracy gains persist for α2 values 0.05, 0.1, and 0.5; the method matches DisturbLabel.Source: §6.2 · p. 7, Table 3 · p. 8

6.4. Evaluation on Face Datasets

  • On face benchmarks, the single model improves over baseline and is comparable to current methods using private data or model ensembles.A two-model ensemble reaches accuracies of 99.31%, 94.43%, 82.50%, 94.88%, and 95.37% in Table 5's protocol order.Source: Table 5 · p. 8, §6.4 · p. 8

7. Conclusion

The conclusion presents Noisy Softmax as a drop-in replacement that postpones early individual saturation and improves CNN performance and generalization. The authors report state-of-the-art or competitive results on several datasets.

7. Conclusion

  • Noisy Softmax injects annealed noise into the softmax input to postpone early individual saturation and improve CNN generalization.The conclusion attributes an effect on parameter updates during back-propagation to early desaturation.Source: §7 · p. 8

Related papers

Loading 1708.03769v1…