PaperlayerView exact revision on arXiv

Source-linked AI summary

Activated Gradients for Deep Neural Networks

Mei Liu, Liangming Chen, Xiaohao Du, Long Jin, Mingsheng Shang

arXiv:2107.04228v1cs.CVcs.AI

TL;DR

Deep neural-network training faces ill-conditioning, vanishing or exploding gradients, and saddle points. The paper applies a gradient activation function to the gradient and analyzes its optimization effects theoretically. Experiments across ImageNet, CIFAR, and PASCAL VOC report improved performance over original models.

  • Problem

    Ill-conditioning, vanishing or exploding gradients, and saddle points remain incompletely resolved challenges in deep neural-network training.

    Source: §I · p. 1

  • Method

    The paper proposes embedding a gradient activation function that modifies gradients to address these optimization problems.

    Source: §I · p. 2, A. Description of the GAF · p. 3

  • Results

    Theoretical analyses establish GAF benefits under stated conditions, while experiments on ImageNet, CIFAR, and PASCAL VOC report significantly better performance than original models.

    Source: §I · p. 2, C. Setup of Experiments · p. 9, D. Comparison Experiments · p. 10

  • Takeaways & Limitations

    GAF can be adopted in various deep neural networks to improve their performance, with implementation requiring roughly one line of code.

    Source: §V · p. 11

Abstract

from arXiv · show

Deep neural networks often suffer from poor performance or even training failure due to the ill-conditioned problem, the vanishing/exploding gradient problem, and the saddle point problem. In this paper, a novel method by acting the gradient activation function (GAF) on the gradient is proposed to handle these challenges. Intuitively, the GAF enlarges the tiny gradients and restricts the large gradient. Theoretically, this paper gives conditions that the GAF needs to meet, and on this basis, proves that the GAF alleviates the problems mentioned above. In addition, this paper proves that the convergence rate of SGD with the GAF is faster than that without the GAF under some assumptions. Furthermore, experiments on CIFAR, ImageNet, and PASCAL visual object classes confirm the GAF's effectiveness. The experimental results also demonstrate that the proposed method is able to be adopted in various deep neural networks to improve their performance. The source code is publicly available at https://github.com/LongJin-lab/Activated-Gradients-for-Deep-Neural-Networks.

I. INTRODUCTION

Deep neural-network training is hindered by ill-conditioning, vanishing or exploding gradients, saddle points, and plateaus. The paper proposes GAF to address these problems and reports theoretical and empirical effectiveness.

Challenges

  • Ill-conditioned optimization creates oscillations in sharp directions, slow convergence in flat directions, and sensitivity to the learning rate.It is characterized by a high condition number and mixed large and small Hessian eigenvalues.Source: §I · p. 1
  • Vanishing and exploding gradients arise through backpropagation when repeated gradient factors drive values toward zero or very large magnitudes.These effects can cause training failure in some layers or divergence, and are partly linked to initialization and activation-function choices.Source: §I · p. 1
  • Saddle points are stationary points with an indefinite Hessian, while high-dimensional models have a high probability of encountering them.The paper also identifies plateaus as flat regions with nearly zero gradients.Source: §I · p. 1

Proposed approach and evidence

  • The proposed gradient activation function acts on gradients to tackle ill-conditioning, vanishing gradients, exploding gradients, and saddle points in one shot.The method is evaluated on ImageNet, CIFAR-10, CIFAR-100, and PASCAL VOC, where consistent or significant improvements are reported.Source: §I · p. 2, §II · p. 3

Prior approaches

  • Existing approaches address these difficulties separately, including Newton’s method, Adam, ReLU, batch normalization, residual structures, gradient clipping, regularization, and noise.The paper notes trade-offs such as Newton’s computational overhead, Adam’s weaker generalization, and gradient clipping’s inability to address vanishing gradients and saddle points.Source: §II · p. 2

III. THEORETICAL ANALYSIS

The paper formalizes GAFs as constrained gradient mappings and develops theoretical analyses of their effect on optimization. It also describes element-wise application to gradient vectors and gives parameterized examples.

A. Description of the GAF

  • A GAF is a second-order differentiable, monotonic increasing, odd function that satisfies boundedness and curvature conditions.The stated conditions include existence of ε ≥ 0 with g ≥ ε implying GAF(g) ≤ g, and g·GAF''(g) < 0.Source: A. Description of the GAF · p. 3
  • When applied to vectors or matrices, the GAF operates element-wise on the gradient.The paper denotes gradient components as g_n(w) = g_n = (∂L/∂w)_n.Source: A. Description of the GAF · p. 3
  • The paper gives log-type GAFs whose shape is controlled by α and β.For the arctan-type GAF, α primarily controls gradient range, while β mainly affects slope near zero once α is fixed.Source: A. Description of the GAF · p. 3

B. Ill-conditioned Problem and Convergence Analysis

The GAF is analyzed under stated smoothness, Lipschitz, convexity, and SGD assumptions. The results show reduced condition numbers and faster expected-loss convergence, while equivalent loss surfaces become less ill-conditioned.

Assumptions

  • The analysis introduces conditions on the loss, gradient, and GAF for studying ill-conditioning and convergence.These include continuously differentiable loss functions, tightly-Lipschitz gradients, strong convexity, and SGD assumptions.Source: B. Ill-conditioned Problem and… · pp. 3–5

GAF properties

  • For small gradient components, a GAF with derivative greater than 1 enlarges differences between transformed gradients.Lemma 1 requires ´g′(gn) > 1 when |gn| ≤ ǫ0.Source: B. Ill-conditioned Problem and… · pp. 3–4
  • For large gradient components, the GAF contracts differences between transformed gradients under its stated derivative and boundedness conditions.Lemma 2 applies when |gn| > ǫ2, assuming ´g′(ǫ1) ≤ 1 and ´g(ǫ2) ≤ ǫ2.Source: B. Ill-conditioned Problem and… · p. 4

Ill-conditioning

  • The GAF reduces the condition number from ζ to ζG, with ζG < ζ.The condition number is defined as ζ = ℓ/c, while the transformed quantities satisfy (ℓ)G < ℓ and the corresponding strong-convexity relation yields the reduction.Source: B. Ill-conditioned Problem and… · p. 5

Convergence

  • SGD with the GAF converges faster in expected loss than original SGD under the stated assumptions, with a linear convergence rate when M = 0.The learning rate is set to η = µ/(ℓMG) under the theorem’s conditions.Source: B. Ill-conditioned Problem and… · pp. 5–6

Equivalent loss surface

  • Equivalent loss surfaces after applying the GAF are significantly less ill-conditioned than the original surface, whereas norm clipping exacerbates ill-conditioning.The comparison uses 3D and contour views; value clipping only slightly suppresses the ill-conditioned problem.Source: B. Ill-conditioned Problem and… · p. 6

C. Vanishing and Exploding Gradient Problems

The paper analyzes how GAFs address vanishing and exploding gradients. For an arctan-type GAF, suitable parameters enlarge sufficiently small gradients, while bounded arctan outputs restrict large gradients.

Setup

  • The subsection uses an arctan-type GAF as a bounded example for analyzing vanishing and exploding gradients.The discussion first gives intuition and then provides formal evidence for the arctan-type choice.Source: C. Vanishing and Exploding Gradient… · p. 6

Vanishing gradients

  • If αβ > 1, sufficiently small nonzero gradients satisfy |´g(gn)| > |gn|.Theorem 5 guarantees an ǫ3 > 0 such that the inequality holds for gn in (0, ǫ3) ∪ (−ǫ3, 0).Source: C. Vanishing and Exploding Gradient… · p. 6
  • The arctan-type GAF enlarges gradients close to zero, addressing the vanishing gradient problem.The paper attributes this behavior to choosing large αβ for small input gradients.Source: C. Vanishing and Exploding Gradient… · p. 6

Exploding gradients

  • Because arctan is bounded, the arctan-type GAF restricts exploding gradients within a certain range.The paper summarizes this boundedness as a solution to the exploding gradient problem.Source: C. Vanishing and Exploding Gradient… · p. 6

D. Saddle Point Problem

The GAF is designed to help optimizers escape saddle-point regions by deforming the equivalent loss surface and enlarging sufficiently small gradients. For an arctan-type GAF, the paper proves faster escape than the original optimizer under stated conditions.

Problem setting

  • The saddle point problem arises at stationary points whose Hessian is indefinite, with negative curvature in at least one dimension.Such curvature gives the optimizer potential to continue minimization, but makes the region difficult to solve.Source: D. Saddle Point Problem · p. 6

Optimization procedure

  • The section frames GAF-equipped gradient descent as an optimizer modification that operates directly on gradients.The implementation is presented as SGDM with GAF and requires α, β, the initial weights, the loss, learning rate, and momentum coefficient.Source: D. Saddle Point Problem · pp. 7–8

Theoretical result

  • An arctan-type GAF escapes a saddle-point region faster than the original gradient descent optimizer when αβ > 1 and gradients remain sufficiently close to zero.Theorem 6 states that suitable ϵ3 > 0 and αβ > 1 yield faster escape over the considered steps.Source: D. Saddle Point Problem · p. 7

Saddle-point mechanism

  • The GAF deforms the loss surface around a saddle point to become steeper, helping the optimizer escape faster.The paper presents this deformation as an alternative view of the GAF’s effect near saddle points.Source: D. Saddle Point Problem · p. 7

IV. EXPERIMENTS

This section introduces the implementation, hyperparameter determination, experiment settings, and comparative experimental results for evaluating GAF-equipped optimizers.

  • The experiments section covers GAF implementation, hyperparameter determination, experiment settings, and comparative experimental results.Source: §IV · p. 8

A. Implementation

GAF is embedded in gradient-based optimizers by activating the computed stochastic gradient before the optimizer update. The implementation adds negligible computational overhead and can be integrated into common deep-learning frameworks.

  • The method first computes the stochastic gradient and then replaces it with its GAF-activated version before the momentum update.SGDM is used as the example implementation; SGD and Adam can apply GAF directly to their gradients.Source: A. Implementation · p. 8
  • GAF can be nested into PyTorch and TensorFlow optimizers with only a few additional code lines and negligible extra calculations.The additional operation is element-wise activation on the gradient.Source: A. Implementation · p. 8
  • 1.2 s of extra training time per epoch was measured on CIFAR-100 with ResNet-50, compared with 42 s for one epoch.Source: A. Implementation · p. 8

B. Hyperparameter Determination

The paper determines GAF hyperparameters from gradient ranges and loss-curve shape, increasing small gradients and restraining large ones to make the optimization landscape more favorable.

  • The recommended process begins by training the original network, recording its maximal gradient, and inspecting the loss curve.These observations guide the choice of GAF type and hyperparameters.Source: B. Hyperparameter Determination · p. 8
  • A quadratic loss curve is considered optimal because gradient descent reaches its minimum in one step with the optimal step size 1/ℓ.Here ℓ is the Lipschitz constant of the gradient.Source: B. Hyperparameter Determination · p. 8
  • For flatter-than-quadratic regions, the GAF should increase the gradient, whereas regions with large gradient Lipschitz constants require restraining large gradients.The latter setting is motivated by observed large-gradient regions in ResNet-18 on CIFAR-10 and DenseNet on CIFAR-100.Source: B. Hyperparameter Determination · pp. 8–9
  • The arctan-type GAF requires αβ > 1 in one case and αβ < 1 in the latter case described by the hyperparameter rules.Source: B. Hyperparameter Determination · p. 9
  • For bounded GAFs such as arctan or tanh, α restricts the maximal gradient and β is chosen so that αβ > 1 for Type 1 loss curves.The paper states that this setting is beneficial for the commonly observed Type 1 shape.Source: B. Hyperparameter Determination · p. 9

C. Setup of Experiments

The experiments compare models with and without GAF across ImageNet, CIFAR-100, CIFAR-10, and PASCAL VOC under specified training settings.

D. Comparison Experiments

Comparison experiments evaluate GAF across multiple architectures, optimizers, datasets, and GAF parameterizations. The reported results consistently show improved performance with GAF.

V. CONCLUSION

The paper concludes that GAF addresses four gradient-based optimization problems through theoretical analysis and improves trained-model performance across several vision benchmarks. It also identifies unresolved directions involving generalization, hyperparameter adaptation, and additional applications.

  • GAF is proposed to ameliorate ill-conditioned, vanishing-gradient, exploding-gradient, and saddle-point problems in one method.Source: §V · pp. 10–11
  • Theoretical analyses demonstrate the feasibility and effectiveness of GAF, whose implementation requires roughly one line of code.Source: §V · p. 11
  • Comparative experiments on ImageNet, CIFAR, and PASCAL VOC report significantly better performance for models equipped with GAF than for original models.Source: §V · p. 11
  • Future work includes studying GAF’s effect on generalization and developing adaptive methods to determine its hyperparameters automatically.Source: §V · p. 11
  • Applying GAF to natural language processing and reinforcement learning tasks is identified as another research direction.Source: §V · p. 11

Related papers

Loading 2107.04228v1…