Source-linked AI summary
Gradient Centralization: A New Optimization Technique for Deep Neural Networks
Hongwei Yong, Jianqiang Huang, Xiansheng Hua, Lei Zhang
TL;DR
DNN training needs optimization methods that are both efficient and generalizable, while existing normalization approaches mainly act on activations or weights. The paper introduces gradient centralization (GC), which centralizes weight gradients and can be viewed as projected gradient descent under a constrained loss. GC is reported to improve optimization stability, efficiency, and generalization across multiple DNN applications.
Problem
DNN optimization must accelerate training and improve generalization, while prior normalization methods mainly operate on activations or weight vectors.
Method
GC subtracts the mean from weight-gradient vectors and applies the centralized gradients within existing gradient-based optimizers.
Results
GC consistently improves learned DNN performance across general and fine-grained image classification, object detection, and instance segmentation.
Takeaways & Limitations
GC offers a simple optimization modification that can regularize weight and output-feature spaces and improve training stability and efficiency.
Takeaways & Limitations
The supplied limitation evidence states only that w0 is the initial weight vector and γ is a scalar, without identifying a broader method limitation.
Abstract
from arXiv · showhide
Optimization techniques are of great importance to effectively and efficiently train a deep neural network (DNN). It has been shown that using the first and second order statistics (e.g., mean and variance) to perform Z-score standardization on network activations or weight vectors, such as batch normalization (BN) and weight standardization (WS), can improve the training performance. Different from these existing methods that mostly operate on activations or weights, we present a new optimization technique, namely gradient centralization (GC), which operates directly on gradients by centralizing the gradient vectors to have zero mean. GC can be viewed as a projected gradient descent method with a constrained loss function. We show that GC can regularize both the weight space and output feature space so that it can boost the generalization performance of DNNs. Moreover, GC improves the Lipschitzness of the loss function and its gradient so that the training process becomes more efficient and stable. GC is very simple to implement and can be easily embedded into existing gradient based DNN optimizers with only one line of code. It can also be directly used to fine-tune the pre-trained DNNs. Our experiments on various applications, including general image classification, fine-grained image classification, detection and segmentation, demonstrate that GC can consistently improve the performance of DNN learning. The code of GC can be found at https://github.com/Yonghongwei/Gradient-Centralization.
1 Introduction
DNN optimizers must accelerate training while improving generalization, but activation- and weight-based normalization methods have limitations. The paper proposes gradient centralization (GC), which directly centralizes gradient vectors and is reported to improve optimization and generalization across applications.
- DNN optimizers aim to reduce the time and cost of reaching good minima while improving test-data prediction accuracy.
- Activation normalization methods such as BN can stabilize and accelerate training, but BN performs poorly with small training batches.
- Weight-based methods such as WN and WS re-parameterize weight vectors, with WS standardizing them to zero mean and unit variance.
- GC operates on gradients by centralizing gradient vectors to zero mean and can be embedded into SGDM and Adam with one line of code.
- GC constrains the loss through a new weight-vector constraint, regularizing weight and output-feature spaces while improving loss Lipschitzness.
- Experiments across image classification, fine-grained classification, detection, and instance segmentation show consistent performance improvements from GC.
2 Related Work
Prior DNN optimization methods operate mainly on activations, weights, or conventional gradient transformations. These methods improve stability, convergence, or generalization, but activation and weight approaches have documented constraints such as sensitivity to batch size or limited scope.
- Activation: Activation normalization includes BN, GN, LN, and IN, with BN normalizing activations across the sample dimension.
- Activation: BN can accelerate training and improve generalization, but its performance degrades when the training batch size is small.
- Weight: WN decouples weight-vector length from direction, while WS standardizes weight vectors and can smooth the loss landscape.
- Gradient: Gradient-based methods include momentum, weight decay, and gradient clipping for acceleration, regularization, or training stability.
3 Gradient Centralization
GC directly modifies weight gradients by subtracting their means rather than applying Z-score normalization. The operation is simple, computationally efficient, compatible with common optimizers, and theoretically linked to smoother optimization.
- BN and WS operate on activations and weight vectors, respectively, whereas GC directly operates on gradients to improve training stability.
- GC computes gradient-vector means and centralizes gradients so each vector has zero mean.
- For fully connected and convolutional layers, GC removes the mean from each weight-gradient column or slice.
- GC can be inserted into SGDM and Adam after gradient computation, including with weight decay, by adding one line of code.
- GC adds negligible computation, requiring 0.6 sec per CIFAR100 ResNet50 epoch compared with 71 sec without that extra time.
4 Properties of GC
GC centralizes gradients and can be understood as projected gradient descent under a weight constraint. This constrains weight space, regularizes output features, smooths optimization, and suppresses large gradients.
- Weight space regularization: GC projects each weight gradient onto a hyperplane before updating the weight, preserving the weight vector’s summed components during training.The projected direction is −P∇w_tL, with e^T w remaining equal to its initial value.
- Weight space regularization: The resulting constrained optimization problem restricts the weight solution space and can reduce overfitting, especially when training samples are limited.
- Output feature space regularization: GC makes output activations less sensitive to constant input-intensity changes when the initial weight vector has a mean close to zero.The activation change depends on γ and the initial weight mean rather than the current weight vector.
- Output feature space regularization: Most convolutional weight-vector means are below e−7 under Kaiming normal or ImageNet pre-trained initialization, supporting this output-feature robustness condition.
- Gradient explosion suppression: GC largely reduces gradient L2 norms and maximum values, helping suppress gradient explosion and making training more stable.The comparison uses ResNet50 trained on CIFAR100, with GC contrasted against training without GC in the first convolutional and fully connected layers.
- Optimization landscape smoothing: GC yields a smoother constrained loss and gradient through improved Lipschitzness, making optimization more predictive and well-behaved.The paper links this smoothing property to faster training, similarly to BN and WS.
5 Experimental Results
Experiments across image classification, detection, and segmentation evaluate GC with varied optimizers, architectures, normalization methods, and hyperparameters. GC consistently accelerates training or improves generalization across these settings.
- GC was evaluated across Mini-ImageNet, CIFAR100, ImageNet, fine-grained classification, object detection, and instance segmentation.The experiments also vary optimizers, architectures, normalization methods, and hyperparameters.
- Mini-ImageNet: On Mini-ImageNet, adding GC to BN accelerated training loss reduction and testing-accuracy growth, and GC also improved BN+WS.The comparison used BN, BN+GC, BN+WS, and BN+WS+GC with ResNet50 and SGDM.
- CIFAR100: GC improved performance across ResNet18, ResNet101, ResNeXt29, VGG11, and DenseNet121 on CIFAR100.This supports GC’s applicability across different DNN architectures.
- CIFAR100: GC consistently improved CIFAR100 performance across weight-decay and learning-rate settings.The experiments used multiple weight-decay values and learning rates for SGDM and Adam.
- On ImageNet, GC improved performance by 0.5% ∼1.2% and largely sped training with GN; on fine-grained datasets, it improved final accuracy by 1.1% ∼2.1%.Detection AP gains were about 0.3% ∼0.6%, while APb increased 0.5% ∼0.9% and APm increased 0.3% ∼0.7%.
- Object Detection and Segmentation: GC improved generalization across all evaluated detection and segmentation models, including models using GN and WS.The reported gains included AP, APb, and APm improvements.
6 Conclusions
The paper concludes that GC is a simple gradient-based optimization method with theoretical and empirical benefits across DNN training settings. It improves training efficiency and generalization across optimizers, architectures, and tasks.
- GC centralizes gradient vectors to zero mean and can be embedded into gradient-based optimizers with one line of code.The method operates directly on gradients rather than activations or weights.
- GC constrains weight vectors, regularizing both weight space and output feature space to improve generalization.The paper analyzes GC as improving the loss function through a weight-vector constraint.
- GC gives the constrained loss function better Lipschitzness and a smoother optimization landscape.The paper links this property to more efficient training and improved gradient behavior.
- Experiments show that GC improves training efficiency and generalization across optimizers, architectures, and multiple DNN tasks.The evaluated tasks include image classification, fine-grained classification, detection, and segmentation.
A1. Proof of Theorem 4.1
The proof establishes a zero-sum property for the projection matrix P and applies it to the SGD update with gradient centralization. It extends the conclusion to momentum-based SGD and considers output activations under shifts by γ1.
- Projection property: The proof shows that 1^T P = 0^T for the projection matrix P.This follows from the stated form P = I − ee^T/M and the definition of M as the dimension of e.
- SGD update: With gradient centralization, each SGD step updates weights using wt+1 = wt − αtP∇wtL.The projection P is applied directly to the gradient in the stated update.
- Output activations: The proof analyzes output activations for x and x + γ1 after the centralized-gradient updates.The supplied passage introduces the activation comparison but does not include its resulting expression.
- Momentum extension: For SGD with momentum, the corresponding momentum term is also zero, so the conclusion remains unchanged.The passage attributes this to the factor γ1^T P and states that the resulting term equals zero.
- Conclusion: The proof concludes after establishing the stated properties for the centralized-gradient update and its momentum variant.
A2. Proof of Theorem 4.2
The proof begins from the fact that e is a unit vector and then applies this property to ΦGC(∇wL). It concludes after this derivation.
- Unit-vector property: The proof uses e^T e = 1 because e is a unit vector.
- Gradient-centralization map: The derivation then evaluates ΦGC(∇wL) using the unit-vector property.
- Conclusion: The proof ends after the stated derivation for ΦGC(∇wL).