Source-linked AI summary
Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H. Nguyen, Madeleine Gibescu, Antonio Liotta
TL;DR
Fully connected neural-network layers create quadratically many parameters and limit model size. SET replaces them with evolving sparse layers, reducing parameters quadratically without decreasing accuracy across three ANN types and 15 datasets.
Problem
Fully connected ANN layers create quadratically many connections, limiting network size, while sparsity is generally pursued only after training.
Method
SET replaces fully connected bipartite layers with sparse layers initialized as Erdős-Rényi graphs and evolved during training toward scale-free connectivity.
Results
SET quadratically reduces parameters without decreasing accuracy, and usually outperforms fully connected counterparts across RBMs, MLPs, and CNNs on 15 datasets.
Takeaways & Limitations
Sparse evolutionary connectivity can replace fully connected ANN layers from the design phase while preserving accuracy and often improving performance.
Takeaways & Limitations
Practical adoption is limited because current GPUs are optimized for dense, while sparse matrix multiplications remain limited in performance.
Abstract
from arXiv · showhide
Through the success of deep learning in various domains, artificial neural networks are currently among the most used artificial intelligence methods. Taking inspiration from the network properties of biological neural networks (e.g. sparsity, scale-freeness), we argue that (contrary to general practice) artificial neural networks, too, should not have fully-connected layers. Here we propose sparse evolutionary training of artificial neural networks, an algorithm which evolves an initial sparse topology (Erdős-Rényi random graph) of two consecutive layers of neurons into a scale-free topology, during learning. Our method replaces artificial neural networks fully-connected layers with sparse ones before training, reducing quadratically the number of parameters, with no decrease in accuracy. We demonstrate our claims on restricted Boltzmann machines, multi-layer perceptrons, and convolutional neural networks for unsupervised and supervised learning on 15 datasets. Our approach has the potential to enable artificial neural networks to scale up beyond what is currently possible.
Introduction
The paper argues that ANNs should replace fully connected layers with sparse connectivity inspired by biological neural networks. It introduces Sparse Evolutionary Training (SET), which adapts sparse bipartite layers during learning while reducing parameters without decreasing accuracy.
- Motivation: Fully connected ANN layers contain most network parameters and create a quadratic number of connections relative to neuron count.This motivates replacing dense layers with sparse topologies.
- Motivation: Biological neural networks exhibit sparse, scale-free, and small-world topologies, whereas ANNs generally do not mimic these features.The paper links this mismatch to the large size of practical ANN models.
- Related work: Unlike fixed-sparsity XBMs, SET is introduced to account for data distributions and address limitations in modeling the data distribution.The preceding XBM approach used a fixed sparse scale-free topology and was reported as faster and more accurate than fully connected RBMs.
- Method: The proposed Sparse Evolutionary Training procedure uses data distributions to create sparse bipartite layers that can replace fully connected bipartite layers in any ANN.SET is inspired by evolutionary approaches to function approximation and network connectivity.
- Contributions: Sparsely connected layers trained with SET can replace fully connected layers without decreasing accuracy while using quadratically fewer parameters from the ANN design phase.The claimed efficiency benefits include reduced connections, memory use, and computational cost before training begins.
Results
SET was evaluated across three ANN types and 15 datasets, reducing parameter counts while maintaining or improving performance. The experiments also showed stable learning and topology evolution toward power-law connectivity in sufficiently large models.
- Evaluation: SET was evaluated on restricted Boltzmann machines, multilayer perceptrons, and convolutional neural networks across 15 benchmark datasets spanning multiple application fields.The evaluation included both unsupervised and supervised learning.
- RBM results: In 7 out of 11 datasets, SET-RBM outperformed fully-connected RBMs while reducing the number of parameters.Table 2 reports the best performer for each model type and dataset.
- RBM results: Fully-connected RBMs exhibited instability and over-fitting, whereas sparse RBM models showed more stable behavior across most cases.On the DNA dataset, fully-connected RBMs peaked after several epochs before declining, and models with more hidden neurons over-fit faster.
- Topology and stability: Hidden-neuron connectivity in SET-RBM and SET-MLP rapidly evolved toward a power-law distribution, with scale-free evolution more consistently observed in larger models.For SET-RBM, cases with fewer neurons failed to evolve to scale-free topologies, while cases with more neurons did so consistently.
- MLP results: SET-MLP outperformed both MLPFixProb and fully-connected MLP while using two orders of magnitude fewer parameters.On CIFAR10, SET-MLP used only 1% of the weights of MLP and achieved significant gains.
- Robustness: SET remained stable with respect to the meta-parameters ε and ζ across the experiments, despite tuning them on only MNIST before evaluation on all 15 datasets.The reported common choice was ζ = 0.3 for SET-RBM and SET-MLP, with model-specific ε values.
Discussion
SET replaces fully connected ANN layers with sparse, evolving topologies and was validated across 15 datasets, three ANN models, and supervised and unsupervised learning. It quadratically reduces parameters without decreasing accuracy, while raising questions about faster topology evolution and large-scale GPU implementation.
- Contributions and validation: SET replaces fully connected bipartite ANN layers with sparse layers and was evaluated on RBMs, MLPs, and CNNs across 15 datasets.The evaluation combined contrastive divergence and stochastic gradient descent for unsupervised and supervised learning.
- Results: Quadratically reducing bipartite-layer parameters caused no decrease in accuracy, while SET models mostly outperformed fully connected counterparts.SET-RBMs, SET-MLPs, and SET-CNNs always outperformed their non-evolutionary counterparts: RBMFixProb, MLPFixProb, and CNNFixProb.
- Network structure and future improvements: SET evolves connections toward scale-free topologies, consistent with real-world complex networks and potentially enabling fewer training epochs through faster topology evolution.The paper identifies preferential attachment algorithms as one possible way to evolve bipartite ANN topologies faster.
- Broader adoption and limitation: SET could extend to recurrent and deep reinforcement learning networks, but large-scale adoption must address the reliance of current implementations on optimized dense GPU matrix multiplication.The discussion frames this implementation issue as an additional step for broader academic and industrial use.
Methods
SET replaces each bipartite fully-connected layer with a sparse Erdős-Rényi layer, trains the network normally, and evolves its connectivity across epochs by pruning and replenishing weights. The procedure is applied to ANN layers during training while preserving the standard training process.
- Sparse Evolutionary Training: SET replaces every bipartite fully-connected layer with a Sparse Connected layer initialized with an Erdős-Rényi topology.The topology is determined by ε and Eq.1.
- Sparse Evolutionary Training: During each training epoch, SET performs the standard training procedure and updates the weights.These operations occur before the sparse-layer evolution step.
- Sparse Evolutionary Training: After weight updates, SET processes each bipartite sparse layer and evolves its connections during training.The pseudocode explicitly iterates over each bipartite Sparse Connected layer after the training and weight-update steps.
- Sparse Evolutionary Training: If the epoch is not the last, SET adds randomly new weights in the same amount as the weights removed previously.This replenishment maintains the number of newly added connections relative to the removed ones.
Materials and Correspondence
The section documents the datasets and experimental setups used to evaluate sparse evolutionary training across RBMs, MLPs, and CNNs, and illustrates the iterative SET connectivity-update procedure.
- SET procedure: SET removes weights closest to zero and randomly adds the same number of new weights after each training epoch.This prune-and-regrow cycle repeats for a finite number of training epochs.
- RBM experiments: RBM experiments vary Contrastive Divergence steps nCD = {1,3,10} and hidden-neuron counts across datasets.The first eight datasets use nh = {100,250,500}, while the last three use nh = {500,2500,5000}.
- Datasets: Experiments cover 15 datasets spanning biology, physics, computer vision, data mining, and economics.The datasets were selected to represent fields where artificial neural networks could advance the state of the art.
- MLP experiments: MLP comparisons differ only in topology: fully connected MLP, fixed Erdős-Rényi sparse MLPFixProb, and SET-trained sparse evolutionary SET-MLP.The experiments report best test-set classification accuracy and model weight counts.