PaperlayerView exact revision on arXiv

Source-linked AI summary

Activated Gradients for Deep Neural Networks

Mei Liu, Liangming Chen, Xiaohao Du, Long Jin, Mingsheng Shang

arXiv:2107.04228v1cs.CVcs.AI

TL;DR

Deep neural-network training faces ill-conditioning, vanishing or exploding gradients, and saddle points. The paper applies a gradient activation function to the gradient and analyzes its optimization effects theoretically. Experiments across ImageNet, CIFAR, and PASCAL VOC report improved performance over original models.

  • Problem

    Ill-conditioning, vanishing or exploding gradients, and saddle points remain incompletely resolved challenges in deep neural-network training.

  • Method

    The paper proposes embedding a gradient activation function that modifies gradients to address these optimization problems.

  • Results

    Theoretical analyses establish GAF benefits under stated conditions, while experiments on ImageNet, CIFAR, and PASCAL VOC report significantly better performance than original models.

  • Takeaways & Limitations

    GAF can be adopted in various deep neural networks to improve their performance, with implementation requiring roughly one line of code.

Abstract

from arXiv · show

Deep neural networks often suffer from poor performance or even training failure due to the ill-conditioned problem, the vanishing/exploding gradient problem, and the saddle point problem. In this paper, a novel method by acting the gradient activation function (GAF) on the gradient is proposed to handle these challenges. Intuitively, the GAF enlarges the tiny gradients and restricts the large gradient. Theoretically, this paper gives conditions that the GAF needs to meet, and on this basis, proves that the GAF alleviates the problems mentioned above. In addition, this paper proves that the convergence rate of SGD with the GAF is faster than that without the GAF under some assumptions. Furthermore, experiments on CIFAR, ImageNet, and PASCAL visual object classes confirm the GAF's effectiveness. The experimental results also demonstrate that the proposed method is able to be adopted in various deep neural networks to improve their performance. The source code is publicly available at https://github.com/LongJin-lab/Activated-Gradients-for-Deep-Neural-Networks.

I. INTRODUCTION

Deep neural-network training is hindered by ill-conditioning, vanishing or exploding gradients, saddle points, and plateaus. The paper proposes GAF to address these problems and reports theoretical and empirical effectiveness.

  • Challenges: Ill-conditioned optimization creates oscillations in sharp directions, slow convergence in flat directions, and sensitivity to the learning rate.It is characterized by a high condition number and mixed large and small Hessian eigenvalues.
  • Challenges: Vanishing and exploding gradients arise through backpropagation when repeated gradient factors drive values toward zero or very large magnitudes.These effects can cause training failure in some layers or divergence, and are partly linked to initialization and activation-function choices.
  • Challenges: Saddle points are stationary points with an indefinite Hessian, while high-dimensional models have a high probability of encountering them.The paper also identifies plateaus as flat regions with nearly zero gradients.
  • Proposed approach and evidence: The proposed gradient activation function acts on gradients to tackle ill-conditioning, vanishing gradients, exploding gradients, and saddle points in one shot.The method is evaluated on ImageNet, CIFAR-10, CIFAR-100, and PASCAL VOC, where consistent or significant improvements are reported.
  • Prior approaches: Existing approaches address these difficulties separately, including Newton’s method, Adam, ReLU, batch normalization, residual structures, gradient clipping, regularization, and noise.The paper notes trade-offs such as Newton’s computational overhead, Adam’s weaker generalization, and gradient clipping’s inability to address vanishing gradients and saddle points.

III. THEORETICAL ANALYSIS

The paper formalizes GAFs as constrained gradient mappings and develops theoretical analyses of their effect on optimization. It also describes element-wise application to gradient vectors and gives parameterized examples.

  • A. Description of the GAF: A GAF is a second-order differentiable, monotonic increasing, odd function that satisfies boundedness and curvature conditions.The stated conditions include existence of ε ≥ 0 with g ≥ ε implying GAF(g) ≤ g, and g·GAF''(g) < 0.
  • A. Description of the GAF: When applied to vectors or matrices, the GAF operates element-wise on the gradient.The paper denotes gradient components as g_n(w) = g_n = (∂L/∂w)_n.
  • A. Description of the GAF: The paper gives log-type GAFs whose shape is controlled by α and β.For the arctan-type GAF, α primarily controls gradient range, while β mainly affects slope near zero once α is fixed.

B. Ill-conditioned Problem and Convergence Analysis

The GAF is analyzed under stated smoothness, Lipschitz, convexity, and SGD assumptions. The results show reduced condition numbers and faster expected-loss convergence, while equivalent loss surfaces become less ill-conditioned.

  • Assumptions: The analysis introduces conditions on the loss, gradient, and GAF for studying ill-conditioning and convergence.These include continuously differentiable loss functions, tightly-Lipschitz gradients, strong convexity, and SGD assumptions.
  • GAF properties: For small gradient components, a GAF with derivative greater than 1 enlarges differences between transformed gradients.Lemma 1 requires ´g′(gn) > 1 when |gn| ≤ ǫ0.
  • GAF properties: For large gradient components, the GAF contracts differences between transformed gradients under its stated derivative and boundedness conditions.Lemma 2 applies when |gn| > ǫ2, assuming ´g′(ǫ1) ≤ 1 and ´g(ǫ2) ≤ ǫ2.
  • Ill-conditioning: The GAF reduces the condition number from ζ to ζG, with ζG < ζ.The condition number is defined as ζ = ℓ/c, while the transformed quantities satisfy (ℓ)G < ℓ and the corresponding strong-convexity relation yields the reduction.
  • Convergence: SGD with the GAF converges faster in expected loss than original SGD under the stated assumptions, with a linear convergence rate when M = 0.The learning rate is set to η = µ/(ℓMG) under the theorem’s conditions.
  • Equivalent loss surface: Equivalent loss surfaces after applying the GAF are significantly less ill-conditioned than the original surface, whereas norm clipping exacerbates ill-conditioning.The comparison uses 3D and contour views; value clipping only slightly suppresses the ill-conditioned problem.

C. Vanishing and Exploding Gradient Problems

The paper analyzes how GAFs address vanishing and exploding gradients. For an arctan-type GAF, suitable parameters enlarge sufficiently small gradients, while bounded arctan outputs restrict large gradients.

  • Setup: The subsection uses an arctan-type GAF as a bounded example for analyzing vanishing and exploding gradients.The discussion first gives intuition and then provides formal evidence for the arctan-type choice.
  • Vanishing gradients: If αβ > 1, sufficiently small nonzero gradients satisfy |´g(gn)| > |gn|.Theorem 5 guarantees an ǫ3 > 0 such that the inequality holds for gn in (0, ǫ3) ∪ (−ǫ3, 0).
  • Vanishing gradients: The arctan-type GAF enlarges gradients close to zero, addressing the vanishing gradient problem.The paper attributes this behavior to choosing large αβ for small input gradients.
  • Exploding gradients: Because arctan is bounded, the arctan-type GAF restricts exploding gradients within a certain range.The paper summarizes this boundedness as a solution to the exploding gradient problem.

D. Saddle Point Problem

The GAF is designed to help optimizers escape saddle-point regions by deforming the equivalent loss surface and enlarging sufficiently small gradients. For an arctan-type GAF, the paper proves faster escape than the original optimizer under stated conditions.

  • Problem setting: The saddle point problem arises at stationary points whose Hessian is indefinite, with negative curvature in at least one dimension.Such curvature gives the optimizer potential to continue minimization, but makes the region difficult to solve.
  • Optimization procedure: The section frames GAF-equipped gradient descent as an optimizer modification that operates directly on gradients.The implementation is presented as SGDM with GAF and requires α, β, the initial weights, the loss, learning rate, and momentum coefficient.
  • Theoretical result: An arctan-type GAF escapes a saddle-point region faster than the original gradient descent optimizer when αβ > 1 and gradients remain sufficiently close to zero.Theorem 6 states that suitable ϵ3 > 0 and αβ > 1 yield faster escape over the considered steps.
  • Saddle-point mechanism: The GAF deforms the loss surface around a saddle point to become steeper, helping the optimizer escape faster.The paper presents this deformation as an alternative view of the GAF’s effect near saddle points.

IV. EXPERIMENTS

This section introduces the implementation, hyperparameter determination, experiment settings, and comparative experimental results for evaluating GAF-equipped optimizers.

  • The experiments section covers GAF implementation, hyperparameter determination, experiment settings, and comparative experimental results.

A. Implementation

GAF is embedded in gradient-based optimizers by activating the computed stochastic gradient before the optimizer update. The implementation adds negligible computational overhead and can be integrated into common deep-learning frameworks.

  • The method first computes the stochastic gradient and then replaces it with its GAF-activated version before the momentum update.SGDM is used as the example implementation; SGD and Adam can apply GAF directly to their gradients.
  • GAF can be nested into PyTorch and TensorFlow optimizers with only a few additional code lines and negligible extra calculations.The additional operation is element-wise activation on the gradient.
  • 1.2 s of extra training time per epoch was measured on CIFAR-100 with ResNet-50, compared with 42 s for one epoch.

B. Hyperparameter Determination

The paper determines GAF hyperparameters from gradient ranges and loss-curve shape, increasing small gradients and restraining large ones to make the optimization landscape more favorable.

  • The recommended process begins by training the original network, recording its maximal gradient, and inspecting the loss curve.These observations guide the choice of GAF type and hyperparameters.
  • A quadratic loss curve is considered optimal because gradient descent reaches its minimum in one step with the optimal step size 1/ℓ.Here ℓ is the Lipschitz constant of the gradient.
  • For flatter-than-quadratic regions, the GAF should increase the gradient, whereas regions with large gradient Lipschitz constants require restraining large gradients.The latter setting is motivated by observed large-gradient regions in ResNet-18 on CIFAR-10 and DenseNet on CIFAR-100.
  • The arctan-type GAF requires αβ > 1 in one case and αβ < 1 in the latter case described by the hyperparameter rules.
  • For bounded GAFs such as arctan or tanh, α restricts the maximal gradient and β is chosen so that αβ > 1 for Type 1 loss curves.The paper states that this setting is beneficial for the commonly observed Type 1 shape.

C. Setup of Experiments

The experiments compare models with and without GAF across ImageNet, CIFAR-100, CIFAR-10, and PASCAL VOC under specified training settings.

  • The comparison experiments use ImageNet, CIFAR-100, CIFAR-10, and PASCAL VOC datasets.
  • ImageNet validation uses single-crop 224 × 224 testing with 1.28 million training images and 50,000 validation images.
  • ImageNet models use mixed-precision training for 150 epochs with batch size 64.

D. Comparison Experiments

Comparison experiments evaluate GAF across multiple architectures, optimizers, datasets, and GAF parameterizations. The reported results consistently show improved performance with GAF.

  • ImageNet comparisons evaluate six ResNet and SE-ResNet architectures using the original method, two gradient-clipping methods, and GAF.
  • The ImageNet comparison results are presented as test accuracies with and without GAF.
  • All tested GAF types improve the performance of the involved CIFAR-100 models under SGDM.Two evaluated parameter schemes are (α = 0.1, β = 20) and arctan (α = 0.2, β = 10).
  • GAF improves performance with Adam on CIFAR-10 and CIFAR-100.
  • GAF improves mAP for SSD300 on PASCAL VOC.

V. CONCLUSION

The paper concludes that GAF addresses four gradient-based optimization problems through theoretical analysis and improves trained-model performance across several vision benchmarks. It also identifies unresolved directions involving generalization, hyperparameter adaptation, and additional applications.

  • GAF is proposed to ameliorate ill-conditioned, vanishing-gradient, exploding-gradient, and saddle-point problems in one method.
  • Theoretical analyses demonstrate the feasibility and effectiveness of GAF, whose implementation requires roughly one line of code.
  • Comparative experiments on ImageNet, CIFAR, and PASCAL VOC report significantly better performance for models equipped with GAF than for original models.
  • Future work includes studying GAF’s effect on generalization and developing adaptive methods to determine its hyperparameters automatically.
  • Applying GAF to natural language processing and reinforcement learning tasks is identified as another research direction.
Loading 2107.04228v1…