Source-linked AI summary
SpinalNet: Deep Neural Network with Gradual Input
H M Dipu Kabir, Moloud Abdar, Seyed Mohammad Jafar Jalali, Abbas Khosravi, Amir F Atiya, Saeid Nahavandi, Dipti Srinivasan
TL;DR
DNNs achieve strong performance but impose high computational costs, motivating a more efficient architecture. The paper introduces SpinalNet, which processes inputs gradually through split layers and evaluates it across regression and classification tasks. The reported results show lower computation and state-of-the-art performance in several benchmark datasets.
Problem
DNNs achieve state-of-the-art performance but still suffer from high computational overhead during training and execution.
Method
SpinalNet splits each layer into input, intermediate, and output components that process partial inputs gradually and can replace fully connected or classification layers.
Results
35.5% reduction in multiplications was reported for one regression setting, while SpinalNet also achieved state-of-the-art performance in several benchmark datasets.
Takeaways & Limitations
SpinalNet is usually less computation-intensive than its counterpart and achieved state-of-the-art results in several handwritten-character and color-image datasets.
Abstract
from arXiv · showhide
Deep neural networks (DNNs) have achieved the state of the art performance in numerous fields. However, DNNs need high computation times, and people always expect better performance in a lower computation. Therefore, we study the human somatosensory system and design a neural network (SpinalNet) to achieve higher accuracy with fewer computations. Hidden layers in traditional NNs receive inputs in the previous layer, apply activation function, and then transfer the outcomes to the next layer. In the proposed SpinalNet, each layer is split into three splits: 1) input split, 2) intermediate split, and 3) output split. Input split of each layer receives a part of the inputs. The intermediate split of each layer receives outputs of the intermediate split of the previous layer and outputs of the input split of the current layer. The number of incoming weights becomes significantly lower than traditional DNNs. The SpinalNet can also be used as the fully connected or classification layer of DNN and supports both traditional learning and transfer learning. We observe significant error reductions with lower computational costs in most of the DNNs. Traditional learning on the VGG-5 network with SpinalNet classification layers provided the state-of-the-art (SOTA) performance on QMNIST, Kuzushiji-MNIST, EMNIST (Letters, Digits, and Balanced) datasets. Traditional learning with ImageNet pre-trained initial weights and SpinalNet classification layers provided the SOTA performance on STL-10, Fruits 360, Bird225, and Caltech-101 datasets. The scripts of the proposed SpinalNet are available at the following link: https://github.com/dipuk0506/SpinalNet
I. INTRODUCTION
The paper proposes SpinalNet to address DNN computational overhead and training difficulties by using gradual inputs inspired by the human somatosensory system. Its experiments report improved performance with fewer computations across several benchmark settings.
- Motivation: DNNs offer state-of-the-art performance but can require large computation and suffer from vanishing gradients during training.Large first hidden layers increase weights, while deep networks can produce negligible gradients near the inputs.
- Motivation: The human spinal architecture and prior biologically inspired networks motivate a neural network with gradual inputs.The paper draws on gradual sensory processing in the spinal cord and the success of CNNs inspired by visual cortex functionality.
- Contribution: SpinalNet is proposed to improve performance with smaller computational overhead by repeatedly receiving partial inputs.The paper investigates SpinalNet as a fully connected layer and reports state-of-the-art results across multiple datasets.
- Study scope: The paper studies SpinalNet for traditional learning, transferred initialization, regression, and classification across more than seventeen datasets.Experiments include VGG-5 and several image datasets, with scripts released for future researchers.
II. THEORETICAL BACKGROUND
SpinalNet adapts characteristics of the human spinal cord into a neural architecture that receives inputs gradually and produces local and global outputs. Its split-layer design keeps inputs and intermediate neurons small while allowing repeated information flow.
- Scope: The architecture is presented as applicable to both shallow neural networks and the fully connected parts of popular deep CNNs.The motivation includes replacing traditional hidden or classifier layers where feature-output relationships may be nonlinear.
- Biological motivation: The proposed architecture mimics gradual input, local output, global influence, and training-dependent weight reconfiguration.These features are presented as similarities between SpinalNet and the human spinal cord.
- Proposed SpinalNet: Each SpinalNet layer contains input, intermediate, and output sub-layers, with partial inputs sent to multiple hidden layers.Intermediate neurons can be changed by the user, while layer inputs and intermediate widths are typically kept small to reduce multiplications.
- Proposed SpinalNet: Successive intermediate sub-layers receive outputs from previous intermediate sub-layers, enabling repeated input processing across layers.The repeated input is intended to compensate for the narrow shape created by small per-layer widths.
C. Universal Approximation of the Proposed SpinalNet
The paper argues that deep SpinalNet retains universal approximation by showing that it can represent a sufficiently wide single-hidden-layer network. It also describes transferred initialization as a training strategy for pretrained networks.
- Universal approximation: The universal-approximation argument reduces the problem to equivalence with a sufficiently wide single-hidden-layer neural network.A wide single hidden layer is treated as a universal approximator, so equivalent deep SpinalNet structures inherit that property.
- Universal approximation: A simplified SpinalNet can be configured to act like a single hidden layer by controlling connections and weights.The visual construction uses simplified layers and zero weights to remove selected cross-layer and output connections.
- Universal approximation: A deep SpinalNet can be equivalent to a single-hidden-layer network with many neurons, supporting universal approximability.The paper presents this equivalence as the basis for its universal-approximation conclusion.
- Transferred initialization: Transferred initialization uses pretrained initial weights while continuing traditional training without freezing convolutional layers.The paper contrasts this with freezing weights, which it says can constrain optimization and fail to propagate important target-dataset features.
III. RESULTS
Experiments evaluate SpinalNet on regression and image classification tasks against traditional and existing networks. Results report lower computation and improved or competitive performance in several settings.
- Regression: 35.5% reduction in multiplications was obtained for the regression SpinalNet while using 14.3k rather than 22k parameters.The traditional network used 21.7k multiplications and 22k parameters; SpinalNet used 14k multiplications and 14.3k parameters.
- Regression: SpinalNet performed better in six of eight regression MSE comparisons.The experiments compared traditional feed-forward and SpinalNet models with the same 300 hidden neurons.
- Experimental design: The study trains multiple networks and SpinalNet variants on seven image-classification datasets using random initialization.Each model was trained ten times, with average and best results reported.
- MNIST: 98.44% accuracy was achieved by CNN with Spinal FC, compared with 98.17% for the default CNN on MNIST.The Spinal FC also reduced fully connected-layer multiplications by more than 48.5% and activation functions by 4%.
- MNIST: 99.72% accuracy was achieved with VGG-5 and Spinal FC on MNIST, described as near state-of-the-art performance.The paper reports this among the top twenty reported accuracies according to its literature search.
2) Fashion-MNIST:
Across Fashion-MNIST, Kuzushiji-MNIST, and QMNIST, SpinalNet classification layers matched or exceeded conventional baselines while reducing multiplication costs in reported comparisons.
- Fashion-MNIST: 94.68% accuracy with VGG-5 (Spinal FC) exceeded the default VGG-5's 94.63% while using fewer multiplications.Random rotation and random resized crops were used for data enhancement.
- Kuzushiji-MNIST: 99.15% accuracy with VGG-5 (Spinal FC) surpassed the default VGG-5's 98.94% and was reported as a new SOTA for Kuzushiji-MNIST.Both models used the same data augmentation, while the Spinal FC used fewer multiplications.
- QMNIST: CNN(Spinal FC) achieved 97.97% and 98.07% QMNIST accuracy with spinal-layer sizes eight and ten, respectively, versus 97.82% for the default PyTorch CNN.The dataset contains fifty thousand test images and shares MNIST's input and output dimensions.
- QMNIST: 99.68% accuracy with VGG-5 (Spinal FC) slightly exceeded VGG-5's 99.66% and was reported as SOTA for QMNIST.Random perspective and random rotation were used to obtain these results.
5) EMNIST:
On EMNIST digits, letters, and balanced datasets, VGG-5 with Spinal FC produced marginally higher accuracy than VGG-5 and was reported as SOTA for each dataset.
- EMNIST digits: 99.82% accuracy with VGG-5 (Spinal FC) exceeded VGG-5's 99.81% and the default PyTorch CNN's 98.89% on EMNIST digits.The EMNIST digits dataset has ten classes and uses 28×28 grayscale images.
- EMNIST letters: 95.88% accuracy with VGG-5 (Spinal FC) exceeded VGG-5's 95.86% and the default PyTorch CNN's 87.57% on EMNIST letters.The result was reported as SOTA for EMNIST letters.
- EMNIST balanced: 91.05% accuracy with VGG-5 (Spinal FC) exceeded VGG-5's 91.04% and the default PyTorch CNN's 79.61% on EMNIST balanced.The result was reported as SOTA, using random perspective and random rotation across EMNIST datasets.
- CIFAR-10 comparison: On CIFAR-10, results did not improve with ResNet-18, whereas VGG-19 achieved higher accuracy with fewer parameters using SpinalNet.The comparison was reported for traditional training on CIFAR-10.
- CIFAR-100 comparison: With VGG-19, replacing two 4096-neuron hidden layers by four 512-neuron spinal layers reduced fully connected-layer multiplications to 7%.The Spinal FC layer also reduced the number of fully connected neurons to half.
- ResNet limitation: The Spinal fully connected layer did not improve ResNet performance, possibly because additional spinal layers decreased gradients in ResNet's initial layers.This limitation was reported for the ResNet setting.
C. Classification: Learning from Transferred Initialization
Using transferred initialization with pretrained VGG-19 and Wide-ResNet models, the study compared traditional and Spinal fully connected layers across CIFAR datasets.
- CIFAR-10: Pretrained VGG-19 bn and Wide-ResNet-101 models were trained on CIFAR-10 with both traditional and Spinal FC layers.The transferred-initialization experiments compared the two classifier designs.
- CIFAR-10: On CIFAR-10, SpinalNet significantly improved VGG-network accuracy while reducing the number of parameters.Wide-ResNet with Spinal FC showed slightly lower performance than the original Wide-ResNet.
- CIFAR-100: On CIFAR-100, a wide Spinal FC with 512 neurons and dropout achieved 88.34% accuracy on Wide-ResNet-101.The result was described as one of the top ten reported CIFAR-100 results.
3) Caltech-101:
SpinalNet performs well on Caltech-101 with both VGG and Wide-ResNet, achieving SOTA performance, while related results indicate accuracy and parameter benefits can depend on the network and dataset.
- SpinalNet performs well with both VGG and Wide-ResNet on Caltech-101 and achieves SOTA performance.
- SpinalNet also performs well on Bird225 with both VGG and Wide-ResNet and achieves SOTA performance.The dataset provider added images and new classes.
- Transferred initialization with Spinal FC produced a slightly better result for both networks but did not reach the top twenty performances.The authors suggest different pre-training data may limit performance.
8) STL-10:
Across the investigated classification settings, SpinalNet achieves strong results with VGG and Wide-ResNet, including SOTA outcomes and several high reported accuracies, while some comparisons remain dataset- or model-dependent.
- SpinalNet achieves SOTA performance on STL-10 with both VGG and Wide-ResNet.
- Wide-ResNet-101 with Spinal FC achieves 100% accuracy on Fruits 360.The passage notes that another paper reported the same efficiency that year.
- SpinalNet achieves 93.77% accuracy on Intel Image Classification and 99.26% on 10 Monkey Species with Wide-ResNet101 and SpinalNet fully connected layers.
D. SOTA Performances
The paper reports SOTA or near-SOTA results across several investigated datasets, while emphasizing that some MNIST gains reflect limited prior investigation rather than the proposed network alone.
- Combining SpinalNet with VGG-5 provides near-SOTA or SOTA performance on MNIST datasets and SOTA performance on five MNIST datasets.
- The authors state that the main reason for the MNIST SOTA results is that few researchers had investigated these newer datasets.
- Transferred initialization with a spinal fully connected layer achieves SOTA performance on several color image datasets.
- SpinalNet computation increases linearly with depth, and the network produces outputs at every layer.
C. Spinal Hidden Layer
SpinalNet can operate independently, replace a traditional wide hidden layer, or serve as a CNN fully connected layer, with configurable segmentation and sub-layer size.
- SpinalNet can function as an independent network, a CNN fully connected layer, or a replacement for a wide traditional hidden layer.
- Its hidden-layer design divides inputs into segments processed across sub-layers, with the number of segments, neurons, and sub-layers configurable.
- Future work includes applying SpinalNet to new datasets, applications, and combinations with other networks.
- The model is motivated by chordate nervous-system connectivity and gradual input processing with local decisions.
- The paper reports that SpinalNet improves classification accuracy and regression error while usually requiring less computation than its counterpart.