Source-linked AI summary
Learning towards Minimum Hyperspherical Energy
Weiyang Liu, Rongmei Lin, Zhen Liu, Lixin Liu, Zhiding Yu, Bo Dai, Le Song
TL;DR
Over-parameterized networks can contain redundant, highly correlated neurons, creating a need for principled diversity regularization. The paper introduces minimum hyperspherical energy (MHE), inspired by the Thomson problem, to distribute normalized neuron directions over a hypersphere, with variants for hidden and output layers. Across image classification, class-imbalance learning, and multiple architectures, MHE consistently improves reported generalization, while its normalization assumes equal neuron importance.
Problem
Over-parameterization can create redundant, highly correlated neurons that may hurt generalization and incur unnecessary computation, while a unified principled theory for optimal compression and generalization remains open.
Method
MHE minimizes pairwise hyperspherical energy of normalized neuron weights, with half-space and angular variants tailored to hidden-layer collinearity and geodesic angular diversity.
Results
MHE consistently improves generalization across generic image classification, class-imbalance learning, large-scale face recognition, multiple architectures, and image generation.
Takeaways & Limitations
MHE is a plug-in regularizer that promotes hyperspherical uniformity without modifying the original network architecture and at negligible computational cost.
Takeaways & Limitations
MHE assumes normalized neurons have equal importance, and incorporating unnormalized norms can create a trivial norm-increase direction and conflict with weight decay.
Abstract
from arXiv · showhide
Neural networks are a powerful class of nonlinear functions that can be trained end-to-end on various applications. While the over-parametrization nature in many neural networks renders the ability to fit complex functions and the strong representation power to handle challenging tasks, it also leads to highly correlated neurons that can hurt the generalization ability and incur unnecessary computation cost. As a result, how to regularize the network to avoid undesired representation redundancy becomes an important issue. To this end, we draw inspiration from a well-known problem in physics -- Thomson problem, where one seeks to find a state that distributes N electrons on a unit sphere as evenly as possible with minimum potential energy. In light of this intuition, we reduce the redundancy regularization problem to generic energy minimization, and propose a minimum hyperspherical energy (MHE) objective as generic regularization for neural networks. We also propose a few novel variants of MHE, and provide some insights from a theoretical point of view. Finally, we apply neural networks with MHE regularization to several challenging tasks. Extensive experiments demonstrate the effectiveness of our intuition, by showing the superior performance with MHE regularization.
1 Introduction
Over-parameterization can create redundant, highly correlated neurons, motivating a principled regularizer that promotes globally diverse neuron configurations. The paper proposes MHE, inspired by the Thomson problem, and introduces half-space MHE for hidden layers.
- Over-parameterization can produce redundant, highly correlated neurons that motivate compression, parameter-efficient architectures, and diversity regularization.
- Existing angular and orthogonality-based methods do not model diversity as global equidistribution of embeddings on a hypersphere.
- MHE defines hyperspherical potential energy from pairwise neuron relationships, where lower energy indicates more diverse and uniformly spaced neurons.
- MHE minimizes hyperspherical energy in each layer, consistently improving neural-network generalization across multiple tasks.
- Half-space MHE adds virtual neurons to prevent opposite-direction collinearity in hidden layers, while output-layer MHE distributes classifier neurons across the full space.
2 Related Works
Prior diversity regularization spans several learning settings, while neural-network methods mainly promote angles, orthogonality, or reduced covariance between bases.
- Diversity regularization has been used in sparse coding, ensemble learning, self-paced learning, metric learning, and neural networks.
- Neural-network diversity methods commonly promote large angles or orthogonality, or reduce covariance between bases.
- Classifier neurons are the projection bases of the last layer before the softmax.
3 Learning Neurons towards Minimum Hyperspherical Energy
MHE regularizes normalized neuron directions by minimizing pairwise hyperspherical energy, with variants for hidden layers, output layers, half-space configurations, and angular distances. It complements weight decay and differs from local orthogonality constraints by targeting global hyperspherical diversity.
- 3.1 Formulation of Minimum Hyperspherical Energy: MHE defines energy over normalized neuron weights on a unit hypersphere and minimizes that energy by varying neuron orientations.
- 3.1 Formulation of Minimum Hyperspherical Energy: The paper uses Riesz s-kernels, with logarithmic energy as the s→0 limiting case; Thomson’s problem corresponds to minimizing E1.
- 3.2 MHE Regularization: MHE is incorporated into the supervised objective alongside training-data fitting, hidden-layer energy, and output-layer energy.
- 3.3 MHE for Hidden and Output Layers: For hidden layers, MHE encourages normalized neurons to be more discriminative and less redundant, while output-layer MHE promotes inter-class feature separability.
- 3.4 MHE in Half Space: Half-space MHE adds an opposite virtual neuron for each hidden-layer neuron, minimizing energy over 2Ni neurons to reduce collinearity redundancy.
- 3.5 MHE beyond Euclidean Distance: Angular MHE replaces Euclidean distance with geodesic distance, equivalently the angle between normalized neurons, but has different optimization dynamics and higher computational cost.
- 3.7 Discussions: MHE promotes angular diversity rather than necessarily enforcing orthogonality, and complements weight decay by regularizing direction rather than kernel magnitude.
- 3.7 Discussions: Normalized neurons avoid trivial norm-increase solutions and prevent conflict between MHE and weight decay, while assuming equal neuron importance.
4 Theoretical Insights
The theory connects minimum hyperspherical energy with asymptotically uniform neuron distributions and explains how energy growth and neuron diversity relate to optimization and generalization.
- Asymptotic behavior: As the number of neurons grows, minimum-energy configurations become uniformly distributed on the hypersphere.
- Asymptotic behavior: For 0<s<d, s=d, and s>d, minimal energy grows respectively as N^2, N^2 log N, and N^(1+s/d).
- Asymptotic behavior: Larger potential power s shifts MHE toward local neighborhood influence rather than global influence from all neurons.
- Generalization intuition: Neuron diversity is linked to lower bounds on a matrix eigenvalue that controls training error, after which generalization error can be bounded using Rademacher complexity.
- Generalization intuition: The theoretical discussion interprets MHE as a diversity-promoting regularizer aligned with analyses showing that neuron diversity can eliminate spurious local minima in one-hidden-layer networks.
5 Applications and Experiments
Experiments evaluate MHE variants across CNN widths and depths, ResNets, ImageNet, class-imbalanced learning, and face recognition. MHE generally improves over baseline methods, with half-space MHE often providing stronger generalization.
- Ablation Study and Exploratory Experiments: MHE variants consistently outperform the baseline on CIFAR-10 and CIFAR-100, with half-space MHE providing the most significant gain.Euclidean and angular distances perform similarly, while s=2 performs best among the tested powers.
- Ablation Study and Exploratory Experiments: MHE and half-space MHE outperform the baseline across network widths, with larger gains when filters are smaller and substantial margins at very wide or narrow widths.Half-space MHE consistently exceeds MHE, indicating the relevance of reducing collinearity redundancy.
- Ablation Study and Exploratory Experiments: MHE enables CNN-15 to converge when the baseline CNN-15 does not, while both MHE variants outperform the baseline across tested network depths.Half-space MHE also generalizes better than MHE at different depths.
- Ablation Study and Exploratory Experiments: Applying MHE to both hidden and output layers yields the best performance in the ablation study, while hidden-layer MHE usually exceeds output-layer MHE.The ablation compares regularization applied to hidden layers, output layers, or both.
- Large-scale Object Recognition: MHE remains effective across hyperparameters from 10^-2 to 10^2 and improves ResNet accuracy without changing the architecture or adding substantial computational cost.Half-space MHE consistently outperforms original MHE across hyperparameter settings, and MHE also improves recognition on ImageNet with ResNet-18 and ResNet-34.
- SphereFace+: Improving Inter-class Feature Separability via MHE for Face Recognition: SphereFace+ combines SphereFace’s intra-class compactness objective with MHE’s inter-class separability objective and achieves state-of-the-art performance on LFW and MegaFace.The paper reports that SphereFace+ achieves the best LFW accuracy and is comparable to the best MegaFace accuracy without model ensemble and PCA.
6 Concluding Remarks
The paper proposes MHE as a plug-in regularizer that encourages hyperspherical diversity of neuron weights and reports consistent improvements across several tasks.
- 6 Concluding Remarks: MHE encourages angular diversity by viewing neuron directions as points distributed uniformly over a hypersphere.It can be applied to every layer without modifying the original network architecture.
- 6 Concluding Remarks: MHE regularization improves results on generic image classification, class-imbalance learning, and large-scale face recognition.The paper also reports significantly improved image generation quality for GANs.
A Experimental Details
The experiments use specified CNN, ResNet, and GAN architectures with task-specific training schedules and SphereFace-compatible face-recognition settings.
- A Experimental Details: CIFAR-10 and CIFAR-100 experiments use batch size 128 and training that terminates after 42.5k iterations.The learning rate starts at 0.1 and is divided by 10 at 20k, 30k, and 37.5k iterations.
- A Experimental Details: ImageNet-2012 experiments use batch size 64, start with learning rate 0.1, and terminate after 500k iterations.The learning rate is divided by 10 at 150k, 300k, and 400k iterations.
- A Experimental Details: CIFAR-10 single-class imbalance removes 90% of class-0 images, while multiple-class imbalance assigns 500×(class_index+1) images per class.These settings contain 45,500 and 27,500 training images, respectively.
- A Experimental Details: SphereFace+ uses the SphereFace preprocessing and testing protocols and applies full-space MHE with Euclidean distance and s = 2 at the output layer.MHE is treated as an additional loss that complements angular softmax by enlarging inter-class angular distance.
- A Experimental Details: Face-recognition results use a two-stage strategy: pretraining with SphereFace followed by finetuning with the proposed loss.The passage states that this strategy is used only for face recognition.
B Proof of Theorem 1 and Theorem 2
The theoretical and methodological material characterizes hyperspherical energy, its minimizing distributions, and how MHE complements magnitude-based regularization and supports weighted neuron control.
- B Proof of Theorem 1 and Theorem 2: The proofs combine results for 0 < s < d, s = d, and s > d to establish Theorem 1 and Theorem 2.The s > d case uses asymptotically optimal point configurations and weak-star convergence results.
- B Proof of Theorem 1 and Theorem 2: For 0 < s < d, hyperspherical energy is minimized by the spherical measure, and optimal configurations become asymptotically uniformly distributed.For s ≥ d, the energy integral becomes infinite and requires separate analysis.
- C Understanding MHE from Decoupled View: MHE regularizes kernel directions, complementing weight decay, which regularizes kernel magnitudes.The decoupled view separates angular effects from norm effects in the convolutional inner product.
- D Weighted MHE: Weighted MHE makes neurons with larger β more fixed and neurons with smaller β more flexible during learning.The experiments vary one or two neuron weights on a three-dimensional sphere.
- E Regularizing SphereNets with MHE: MHE outperforms the SphereNet baseline and orthonormal regularization when regularizing SphereNets.The experiment uses the same architecture and training setting as CNN-9 with cosine SphereConv.
- F Improving AM-Softmax with MHE: Combining MHE with AM-Softmax increases LFW accuracy from 99.26% to 99.37%.The experiment uses SphereFace-20 trained on CASIA-WebFace.
G.1 Network Architecture for GAN
The GAN experiments specify CIFAR-10 generator and discriminator architectures and compare images generated by a baseline GAN with images from GANs regularized by MHE and spectral normalization.
- G.1 Network Architecture for GAN: The generator and discriminator architectures used in the image-generation experiments are detailed in Table 15.The table covers CIFAR-10 architectures for the generator and discriminator.
- G.1 Network Architecture for GAN: For CIFAR-10, the generator begins with a 128-dimensional Gaussian input and uses dense, deconvolutional, batch-normalization, ReLU, convolutional, and Tanh stages.The architecture uses Mg = 4 for CIFAR-10.
- G.1 Network Architecture for GAN: The study compares randomly generated images from a baseline GAN with images from GANs regularized by both MHE and spectral normalization.The generated images are shown in Figure 7.
H.1 Class-imbalance learning on CIFAR-100
On imbalanced CIFAR-100, MHE consistently improves CNN performance, with half-space MHE usually outperforming full-space MHE. The accompanying visualization attributes this to more uniformly distributed classifier neurons.
- MHE consistently improves CNNs in CIFAR-100 class-imbalance learning.
- Half-space MHE performs better than full-space MHE in most imbalanced CIFAR-100 cases.
- Without MHE, imbalanced-data classifier neurons become highly biased toward another class and cannot be properly learned.
- With MHE, classifier neurons become more uniformly distributed in the 2D feature visualization.
I More results of SphereFace+ on Megaface Challenge
Additional SphereFace+ experiments on Megaface show that MHE-enhanced models outperform the SphereFace baseline across tested mSF settings. The section reports both identification-rate and ROC-curve evaluations.
- SphereFace+ consistently outperforms the SphereFace baseline on Megaface verification rate across tested mSF settings.
- MHE enhances verification rate on the Megaface challenge.
- The evaluation includes identification rate versus distractor size and ROC-curve results.
J Training SphereFace+ on MS-Celeb and IMDb-Face datasets
SphereFace+ is further evaluated after training on the larger MS-Celeb and IMDb-Face datasets, extending assessment beyond the smaller CASIA-WebFace training set.
- SphereFace+ is trained on MS-Celeb with 8.6M images and IMDb-Face with 1.7M images.
- The earlier SphereFace+ results used the relatively small CASIA-WebFace dataset containing 0.49M images.
- The models use the SphereFace-20 architecture and are evaluated on LFW verification accuracy and MegaFace rank-1 identification.