PaperlayerView exact revision on arXiv

Source-linked AI summary

diffGrad: An Optimization Method for Convolutional Neural Networks

Shiv Ram Dubey, Soumendu Chakraborty, Swalpa Kumar Roy, Snehasis Mukherjee, Satish Kumar Singh, Bidyut Baran Chaudhuri

arXiv:1909.11015v4cs.LGcs.CVcs.NEmath.OC

TL;DR

Basic SGD does not adapt step sizes to parameter-specific gradient behavior, while existing adaptive methods do not use local gradient changes. The paper proposes diffGrad, which combines Adam with current-to-previous gradient differences, and reports stronger image-categorization performance than the compared optimizers.

  • Problem

    Basic SGD uses equal-sized parameter updates despite differing gradient behavior, while common adaptive methods do not use local gradient changes.

  • Method

    diffGrad extends Adam with current and immediate past gradient differences to control learning rates according to short-term gradient changes.

  • Results

    diffGrad outperforms the compared optimizers on CIFAR10 and CIFAR100 validation classification across tested batch sizes.

  • Takeaways & Limitations

    The paper supports diffGrad as an optimizer evaluated across synthetic non-convex functions and ResNet-based CNN image categorization experiments.

Abstract

from arXiv · show

Stochastic Gradient Decent (SGD) is one of the core techniques behind the success of deep neural networks. The gradient provides information on the direction in which a function has the steepest rate of change. The main problem with basic SGD is to change by equal sized steps for all parameters, irrespective of gradient behavior. Hence, an efficient way of deep network optimization is to make adaptive step sizes for each parameter. Recently, several attempts have been made to improve gradient descent methods such as AdaGrad, AdaDelta, RMSProp and Adam. These methods rely on the square roots of exponential moving averages of squared past gradients. Thus, these methods do not take advantage of local change in gradients. In this paper, a novel optimizer is proposed based on the difference between the present and the immediate past gradient (i.e., diffGrad). In the proposed diffGrad optimization technique, the step size is adjusted for each parameter in such a way that it should have a larger step size for faster gradient changing parameters and a lower step size for lower gradient changing parameters. The convergence analysis is done using the regret bound approach of online learning framework. Rigorous analysis is made in this paper over three synthetic complex non-convex functions. The image categorization experiments are also conducted over the CIFAR10 and CIFAR100 datasets to observe the performance of diffGrad with respect to the state-of-the-art optimizers such as SGDM, AdaGrad, AdaDelta, RMSProp, AMSGrad, and Adam. The residual unit (ResNet) based Convolutional Neural Networks (CNN) architecture is used in the experiments. The experiments show that diffGrad outperforms other optimizers. Also, we show that diffGrad performs uniformly well for training CNN using different activation functions. The source code is made publicly available at https://github.com/shivram1987/diffGrad.

I. INTRODUCTION

The paper motivates adaptive optimization for neural networks and proposes diffGrad, which uses short-term gradient changes to adjust parameter learning rates. It analyzes convergence and evaluates the method on synthetic non-convex functions and image categorization with ResNet-based CNNs.

  • Motivation: Basic SGD updates every parameter with equal learning-rate behavior despite differences in how the loss changes across optimization dimensions.The paper identifies iteration-wise gradient behavior as a source of poor optimization.
  • Proposed method: diffGrad extends Adam by incorporating the difference between current and immediate past gradients for each parameter.This provides local gradient-change information beyond accumulated gradient moments.
  • Proposed method: Large gradient changes trigger higher learning rates, whereas small changes automatically lower the learning rate near likely optima.The method uses short-term gradient behavior to adapt the step size to the optimization stage.
  • Proposed method: Past-gradient accumulation helps diffGrad address saddle-point scenarios alongside its gradient-change-based learning-rate control.The paper connects accumulated moments with handling saddle points.
  • Analysis: The paper derives a regret-bound convergence analysis and conducts empirical analysis on three synthetic complex non-convex functions.It models the optimization problem as a regression problem for the synthetic-function analysis.
  • Evaluation: Experiments evaluate diffGrad with ResNet-based CNNs, different optimizer variants, and different activation functions for image categorization.The study reports improved performance in this experimental setting.

II. PRELIMINARIES

The preliminaries describe SGD and adaptive moment methods, including Adam and AMSGrad, and motivate diffGrad's friction-based control of learning rates. These methods differ in how they use gradients, moments, and accumulated squared gradients to regulate parameter updates.

  • SGD: SGD updates all parameters with the same learning rate α_t in each iteration, using gradients computed from the loss.The update proceeds in the opposite direction of each parameter's gradient.
  • SGDM: SGDM incorporates current and past gradients into a per-dimension moment for parameters with consistent gradient behavior.The accumulated moment supplies momentum for optimization, including when current gradients are near zero.
  • Adaptive methods: AdaGrad normalizes learning rates using the accumulated sum of squared past gradients, while AdaDelta and RMSProp use decaying accumulation.AdaGrad's accumulated squares can make the effective learning rate decrease drastically over iterations.
  • Adam: Adam computes per-parameter learning rates from bias-corrected first and second moments based on gradients and squared gradients.The first and second moments correspond to the mean and variance of gradients.
  • diffGrad motivation: diffGrad introduces a friction coefficient that applies less friction when gradient changes are large and more friction when gradient changes are small.The coefficient is bounded between 0.5 and 1, with low gradient change reducing the learning rate near an optimum.
  • AMSGrad: AMSGrad addresses Adam's low-second-moment overshooting problem by normalizing updates with the maximum second moment across current and past iterations.The stated failure mode is divergence after overshooting an optimum because friction decreases when the second moment becomes small.

III. PROPOSED DIFFGRAD OPTIMIZATION

diffGrad adapts each parameter’s learning rate using short-term gradient changes, adding friction where gradients change slowly and allowing larger steps where they change rapidly. It retains Adam’s moment calculations while introducing a diffGrad friction coefficient to regulate updates and reduce overshooting.

  • diffGrad controls each parameter’s learning rate using the change between immediate past and current gradients.The method computes Adam-like first- and second-order moments, then uses short-term gradient behavior for dynamic adjustment.
  • The diffGrad friction coefficient ξ is computed from the absolute gradient change through a nonlinear sigmoid function.AbsSig maps values between 0.5 and 1, while Δg_t,i measures the change between consecutive gradients.
  • Large gradient changes incur less friction, whereas small changes incur more friction, reaching at most 0.5 when the gradient does not change.Thus, DFC imposes more friction in slowly changing regions and less friction in rapidly changing regions.
  • The update rule combines the learning rate, bias-corrected moments, and DFC to update each parameter while avoiding division by zero.The small ε term is approximately 10^-7, and ξ_t,i is the friction coefficient used in the parameter update.
  • Compared with Adam and AMSGrad, diffGrad uses DFC to regulate first-moment effects, permit larger steps on rapidly changing surfaces, and reduce overshooting on slowly changing surfaces.The minimum DFC is set to 0.5 so optimization can retain sufficient step magnitude in flat local optima and saddle regions.

IV. CONVERGENCE ANALYSIS

The convergence analysis evaluates diffGrad in the online-learning framework using regret bounds under bounded-gradient and bounded-distance assumptions. The resulting bound is comparable to general convex online-learning methods and improves over the bound for nonadaptive methods.

  • The analysis models an unknown sequence of convex cost functions and evaluates diffGrad through the regret bound framework.The parameter sequence is selected iteratively without knowing the future cost functions in advance.
  • The diffGrad regret bound is comparable to general convex online-learning methods.The paper derives the guarantee using definitions of per-parameter gradients and their iteration histories.
  • The theoretical analysis uses a decaying first-moment parameter β1,t, with λ typically close to 1.The decay is used similarly to Adam in the convergence analysis.
  • Adaptive methods such as diffGrad and Adam achieve a dimension-dependent term that improves over the O(dT) bound of nonadaptive methods.The additive term over dimension d can be much smaller than its stated upper bound.
  • Under bounded gradients and bounded parameter distances, theorems and a corollary provide a regret guarantee for diffGrad.The assumptions include bounds G and G∞ on gradients and D and D∞ on distances between iterates.

V. EMPIRICAL ANALYSIS

The empirical analysis compares Adam and diffGrad on three one-dimensional non-convex functions, examining regression loss and parameter trajectories over iterations. diffGrad avoids Adam’s overshoot on F1 and F2 and is more stable around F3’s global minimum.

  • Adam and diffGrad are evaluated as optimizers for a regression problem defined over three one-dimensional non-convex functions.The functions are denoted F1, F2, and F3.
  • F1 has one global and one local minimum, while F2 and F3 each have one global and two local minima.
  • Both methods use β1 = 0.95, β2 = 0.999, η = 0.1, zero-initialized moments, θ = −1, and 300 optimization iterations.Regression loss and θ are recorded at each iteration.
  • On F1, Adam overshoots the global minimum at θ = −0.3 and becomes stuck at the local minimum θ = 0.2.The same behavior is observed for F2.
  • diffGrad’s DFC controls momentum toward the global minimum, preventing overshoot and reaching zero loss where Adam saturates at a nonzero loss.
  • Both methods reach F3’s global minimum, but Adam oscillates more around it, whereas diffGrad obtains better stability.Both optimizers accumulate enough momentum to cross F3’s preceding local minimum.

VI. EXPERIMENTAL SETUP FOR CLASSIFICATION

The classification experiments use an image-categorization setup designed to evaluate optimization methods with a deep convolutional architecture, hyperparameters, and applied datasets.

  • The experimental setup specifies the deep architecture, hyperparameters, and dataset used for image categorization experiments.

A. Deep Architecture Used

The image-categorization experiments use a ResNet-based CNN architecture implemented in PyTorch, with depth 50 and residual bottleneck units.

  • Deep Architecture Used: The experiments use a PyTorch implementation of ResNet50 for CIFAR image categorization.
  • Deep Architecture Used: ResNet residual units connect each unit’s input directly to its output, facilitating training of deeper architectures.
  • Deep Architecture Used: The ResNet50 design includes convolutional layers, batch normalization, repeated bottleneck blocks, and average pooling.The listed bottleneck repetitions are 3, 4, 6, and 3 across successive stages.

B. Hyper-Parameter Setting

The classification experiments compare optimization methods across batch sizes using a 100-epoch schedule and CIFAR datasets with standardized augmentation and validation-accuracy tables.

  • Hyper-Parameter Setting: The classification experiments use batch sizes of 32, 64, and 128 over 100 epochs.The learning rate is 10^-3 for the first 80 epochs and 10^-4 for the final 20 epochs.
  • Hyper-Parameter Setting: SGDM uses momentum coefficient 0.9, while other optimizers use PyTorch’s default settings.
  • Dataset Used: Experiments use CIFAR10 and CIFAR100, each containing 60,000 32 × 32 × 3 images split into 50,000 training and 10,000 validation images.CIFAR10 has 10 categories and CIFAR100 has 100 categories.
  • Dataset Used: Table I compares validation classification accuracy across seven optimizers and batch sizes 32, 64, and 128.
  • Dataset Used: Table II compares validation classification accuracy across variants of diffGrad using batch sizes 32, 64, and 128.
  • Dataset Used: Training images are augmented by horizontal flipping with probability 0.5 and random 32 × 32 crops from zero-padded 40 × 40 images.

VII. CLASSIFICATION EXPERIMENTS AND ANALYSIS

The classification experiments evaluate diffGrad with ResNet50 on CIFAR10 and CIFAR100, compare it with established optimizers, and examine friction-coefficient and activation-function variants. diffGrad outperforms the compared optimizers across tested batch sizes, while variant and activation results depend on dataset, batch size, and activation choice.

  • Validation Results Comparison: diffGrad outperforms SGDM, AdaGrad, AdaDelta, RMSProp, AMSGrad, and Adam on CIFAR10 and CIFAR100 for batch sizes 32, 64, and 128.The reported comparison uses validation classification accuracy with ResNet50.
  • Experiments with diffGrad Variants: The original diffGrad performs better on CIFAR10 at high batch sizes and on CIFAR100 at small batch sizes than its tested DFC variants.The variants modify the diffGrad friction coefficient using absolute gradients, signed differences, or batch statistics.
  • Experiments with diffGrad Variants: The DFC variants include DFC1 without the absolute gradient difference, DFC2 with a rescaled coefficient, and DFC3–DFC5 using batch-gradient mean and standard deviation.Their coefficient ranges differ, including DFC1 ∈ [0, 1], DFC2 ∈ [0.5, 5], and DFC3–DFC5 ∈ [0.5, 1].
  • Performance Analysis with Activation Functions: ELU, ReLU, and LReLU achieve the best reported validation accuracy for batch sizes 32, 64, and 128, respectively, in the CIFAR10 activation-function experiment.The experiment compares ReLU, LReLU, ELU, and SELU with ResNet50.
  • Conclusion: The conclusion reports that diffGrad controls update steps to avoid overshooting and oscillation around the global minimum on three synthetic non-convex functions.The paper also reports testing diffGrad with ResNet50 for image categorization on CIFAR10 and CIFAR100.

A. Convergence Proof

The convergence analysis derives a regret-bound guarantee for diffGrad under bounded-gradient and bounded-parameter-distance assumptions. The proof aggregates dimension-wise inequalities across a sequence of convex functions and uses established Adam lemmas.

  • Convergence Proof: The analysis uses a decaying first-moment parameter β1,t = β1λ^(t−1), with λ ∈ (0, 1) typically close to 1, such as 1 − 10^-8.The convergence statement is presented under this parameterization and the boundedness assumptions.
  • Convergence Proof: The proof applies Adam’s lemmas to the diffGrad update rule while accounting for the friction coefficient ξt,i.Because 0.5 ≤ ξt,i ≤ 1, the coefficient can be removed from one inequality term without violating the bound.
  • Convergence Proof: The regret bound is derived by aggregating the analysis over all parameter dimensions and across the sequence of convex functions.The derivation uses Young’s inequality and a lemma from Adam’s analysis.
  • Convergence Proof: The proof initializes the friction-related quantity using ξ1,i = 1/(1 + e^−|g1,i|), given g0,i = 0, before stating the diffGrad regret bound.This follows from the definition of the initial gradient difference.
Loading 1909.11015v4…