PaperlayerView exact revision on arXiv

Source-linked AI summary

Purin: A Biology-inspired Mechanism for Artificial Neural Networks

Zishu Liu, Chunbo Luo, Christos Grecos

arXiv:2609.31235v1cs.AIcs.NE

TL;DR

ANNs generally use fixed trainable weights during a batch, while biological synapses can change efficacy over short timescales. Purin adds short- and long-term efficacy mechanisms through an ANN-compatible time-interval abstraction. After confounding factors are removed, Purin-equipped matched models achieve higher classification accuracy across the evaluated datasets.

  • Problem

    ANNs typically use fixed trainable weights during a batch, omitting short-term synaptic efficacy changes, while discrete time-step simulation requires extra temporal processing.

    Source: §3.1 · p. 2, From Time Step to Time Interval · p. 3

  • Method

    Purin combines input- and output-side efficacy weights with an activity-dependent temporary factor, using time intervals to model short- and long-term changes without spike encoding or discrete time-step simulation.

    Source: §1 · p. 1, §3 · p. 2, From Time Step to Time Interval · p. 3

  • Results

    After removing confounding factors, Purin-equipped matched models achieved higher classification accuracy on all four datasets across the evaluated architectures.

    Source: §5 · p. 7

  • Takeaways & Limitations

    The split weight mechanism contributed most to OA improvement, while gstp further improved accuracy by approximately 0.2% to 5% in most cases.

    Source: §4.3 · p. 6

    Purin was not evaluated with residual architectures such as ResNet18; extending it to residual connections remains future work.

    Source: §5 · p. 7

Abstract

from arXiv · show

Artificial neural networks (ANNs) usually represent neural transmission with fixed trainable weights during a training batch, which omits short-term changes in synaptic efficacy. In addition, the discrete time-step simulation requires additional temporal processing that many conventional ANN architectures do not use. To overcome these challenges, we propose Purin, a biology-inspired and ANN-compatible mechanism, that introduces synaptic efficacy modulation into conventional convolutional neural networks. Purin uses a time-interval-based abstraction for neural activities, which allows Purin to introduce short- and long-term synaptic efficacy changes without using discrete time-steps. Purin introduces a bounded factor to represent temporary synaptic efficacy changes, together with two weight matrices that represent input-side and output-side efficacy. The weight matrices are updated by backpropagation and interpreted as the long-term synaptic efficacy changes. Experimental results show that after removing the confounding factors in the AlexNet, VGG11, and GoogLeNet architectures, Purin improves the classification accuracies in all three models across the evaluated datasets.

1 Introduction

ANNs differ from biological neurons in several respects, while spiking neural networks face training difficulties because their spike-generation functions are not differentiable. Purin introduces synaptic efficacy mechanisms designed to narrow selected biological gaps while remaining compatible with conventional ANN architectures and backpropagation.

1 Introduction

  • SNNs introduce biologically plausible spiking neurons, but nondifferentiable spike-generation functions prevent direct use of backpropagation.This training difficulty motivates approaches that add biological mechanisms while retaining ANN learning compatibility.Source: §1 · p. 1
  • Purin uses two split weight matrices for input-side and output-side efficacy, with backpropagation updating the matrices as long-term changes.The matrices represent efficacy on the input and output sides of a neuron.Source: §1 · p. 1
  • Purin treats still images as information presented over a short interval and uses activation-dependent gstp to model temporary within-batch weight changes.This time-interval abstraction adds short- and long-term dynamics without extra transformations such as spike encoding.Source: §1 · p. 1

2 Related work

Related work describes biological approaches that make ANN training or image preprocessing more challenging, alongside ANN attention and gating methods for sample-dependent features. Purin is positioned against discrete-time plasticity models and methods that sustainably change trainable parameters.

2 Related work

  • SNNs model spiking neurons, but discrete-spike training is challenging; surrogate gradients and ANN-to-SNN conversion address training, while image spike encoding adds preprocessing.CNN-feature-extractor/SNN combinations still require spike encoding.Source: §2 · pp. 1–2
  • Short-term plasticity models often use multiplicative factors, but the cited models are usually based on spiking or discrete time steps incompatible with continuous ANN inputs or outputs.The passage cites Abbott et al. 1997 and Hennig 2013.Source: §2 · p. 2
  • Attention and gating methods model sample-dependent features, but many sustainably change trainable parameters, unlike biological short-term plasticity.Examples include SE blocks (Hu, Shen, and Sun 2018), coordinate attention (Hou, Zhou, and Feng 2021), LSTM, and gMLP (Liu et al. 2021).Source: §2 · p. 2

3 Method

Purin brings selected features of synaptic efficacy modulation into CNNs while retaining continuous activations and backpropagation. It uses an interval-based activity abstraction, split input- and output-side efficacy weights, and a temporary modulation factor.

3.2 Biology-inspired Abstraction of Neuron

  • A Purin neuron uses Win and Wout for input- and output-side efficacy, f(·) as a signed activity measure, and gstp for temporary efficacy changes.For convolution layers, gstp has one value per kernel and modulates channels.Source: §3 · p. 2

3.1 Motivation

  • Purin narrows selected gaps between ANN and biological neurons without aiming for biophysical fidelity or requiring spike generation, spike encoding, or discrete temporal simulation.It is designed to remain compatible with continuous activations and the BP algorithm.Source: §3.1 · p. 2

Output Efficacy

  • Purin adds non-negative output-side efficacy Wout after activation to represent presynaptic transmission efficacy without reversing the activation’s sign.Purin does not model signal decay during transmission because neuron-to-neuron distance is difficult to define in ANNs.Source: Output Efficacy · pp. 2–3

From Time Step to Time Interval

  • Purin treats an input image as information presented over an abstract interval: Win and Wout stay fixed while gstp changes, then the weights update by BP and gstp resets after a batch.The interval has no specified physical duration; f(·) is interpreted as signed activity relative to a baseline.Source: From Time Step to Time Interval · p. 3

3.3 Short-term Plasticity

  • A bounded factor gstp represents short-term efficacy changes and updates during feedforward processing; in convolution layers, its update uses channel activity.The factor is limited to 0.8–1.2 and recovers toward its default value; it resets after long-term weight updates.Source: §3.3 · p. 4

3.4 Recovery Analysis of Short-term Factor

  • With zero inputs in the recovery analysis, the distance between gstp and its default value shrinks after each recovery step.The analysis presents this as recovery behavior, while the bounded factor prevents the output from growing without bound.Source: §3.4 · p. 4

3.5 Weight Update

  • Win and Wout are trained by gradient descent; for convolution layers, shared kernels make the output-weight update channel-wise.The convolution update uses the output feature map and its backpropagation error signal.Source: §3.5 · p. 4

3.6 The Bias Vector and Dropout Mechanism

  • Purin excludes the bias vector because the authors found no satisfactory interpretation under the interval abstraction, and excludes dropout because its random mask would corrupt activity used to update gstp.Source: §3.6 · pp. 4–5

4 Experiments

Experiments evaluate Purin in AlexNet, VGG11, and GoogLeNet across four image datasets. After matching key benchmark conditions, Purin improves overall accuracy across all four datasets, while ablations and comparisons examine its components and alternatives.

4.1 Datasets and Experimental Settings

  • Experiments use AlexNet, VGG11, and GoogLeNet on CIFAR-100, AID, UCM, and Oxford102, with dataset-specific splits and models trained from scratch.Training from scratch avoids mismatched input-feature distributions from pretrained weights obtained without the split synaptic mechanism.Source: §4 · p. 5, §4.1 · p. 5

4.2 Comparisons with Benchmark Models

  • Purin improves OA over matched models on all four datasets; AlexNet and VGG11 gain approximately 10% on Oxford102 and UCM, and 3% to 7% on AID and CIFAR-100.Matched GoogLeNet accuracy recovers toward vanilla GoogLeNet, including about 24% and 13% OA recovery on UCM and CIFAR-100, respectively.Source: Table 3 · p. 6, §4.2 · p. 6

4.3 Ablation Study

  • Ablations identify split weights as the largest contributor to OA gains; gstp adds approximately 0.2% to 5% accuracy in most cases.The full combination can improve OA over both matched models and split-weight-only variants; architecture-specific differences are small for AlexNet and GoogLeNet.Source: §4.3 · p. 6, Table 5 · p. 6

4.4 Short-term Factor Variants

  • Variant A achieves the lowest OA on both datasets for VGG11 and GoogLeNet; its gap versus other variants is 2–6% for VGG11 and approximately 18% for GoogLeNet.The other four variants have similar OA values, differing by approximately 1%, with variant E achieving the highest accuracy.Source: §4.4 · p. 7, Table 7 · p. 7

4.5 Comparisons with the SE Block

  • On CIFAR-100 and UCM, all three matched GoogLeNet models with or without SE blocks have lower OA than GoogLeNet+Purin.The combination inserting SE after all convolution layers is about 17% to 20% lower on both datasets.Source: Table 8 · p. 7, §4.5 · p. 7

5 Conclusions and Future Work

After confounders are removed, Purin improves classification accuracy across all four datasets for all matched models, though vanilla-model comparisons are not consistent. The paper identifies residual architectures as future work because they were not evaluated.

5 Conclusions and Future Work

  • Purin improves classification accuracy on all four datasets for all matched models after confounding factors are removed.Compared with vanilla baselines without removing those factors, results do not consistently improve across architectures.Source: §5 · p. 7
  • In most cases, the full Purin mechanism further improves accuracy over the split-weight mechanism alone.The exponential recovery and value-range limitation for gstp can achieve higher overall accuracy; comparisons suggest gstp⊙Wout is not a weaker SE-style recalibration module.Source: §5 · p. 7
  • Purin was not evaluated with residual architectures such as ResNet18 because their neural structure differs.Extending Purin to residual connections is proposed as future work; a ResNet20-like example connects the first and last convolution layers within a block.Source: §5 · p. 7, Figure 4 · p. 8

A An Attempt to Combine Purin and Residual Connection

This experiment explores whether Purin can be incorporated into residual connections, rather than testing whether it outperforms vanilla ResNet20. It uses a ResNet20-like model on CIFAR-10 without data augmentation.

A An Attempt to Combine Purin and Residual Connection

A.1 Residual implementation and results

The residual Purin model applies Purin in residual connections and sums outputs from the beginning and end layers. On CIFAR-10, it achieves comparable accuracy to vanilla ResNet20, with a 0.18% drop.

A.1 Residual implementation and results

  • Residual connections use 1 × 1 convolution layers with Purin and sum the beginning- and end-layer outputs.The configuration matches vanilla ResNet20 except that batch normalization layers are omitted.Source: §A.1 · p. 8
  • The Purin residual model’s accuracy is comparable to vanilla ResNet20, with a 0.18% drop.The reported measure is CIFAR-10 accuracy; Table 9 gives averages and standard deviations.Source: Table 9 · p. 8, §A.1 · p. 8
  • The proposed residual model avoids severe accuracy degradation while incorporating Purin into residual connections.Source: §A.1 · p. 8

B Short-term Factor Behavior Visualization

The paper visualizes gstp values for three neurons per layer in a trained Purin+VGG11 model on a CIFAR-100 evaluation batch. The values do not reach either boundary, indicating mild output scaling; the increasing trend is attributed to leaky ReLU.

B Short-term Factor Behavior Visualization

  • In a trained Purin+VGG11 model, the visualization samples three neurons from each convolutional or fully connected layer on a CIFAR-100 evaluation batch.Layers are numbered 0–16, including maxpooling and flatten layers.Source: B Short-term Factor Behavior… · p. 8
  • gstp values do not saturate at either boundary within a batch, so the short-term factor provides mild output scaling.Source: B Short-term Factor Behavior… · p. 8
  • The increasing gstp trend is attributed to leaky ReLU scaling negative outputs by 0.01, weakening their influence on gstp updates.Source: B Short-term Factor Behavior… · p. 8

C Additional Trainable Parameters Overhead Analysis

Purin adds trainable output-side weights to convolutional and fully connected layers, with parameter overhead below 0.2% in all three matched architectures. The final fully connected layer disables the added weights and gstp, so it contributes no additional parameters.

C Additional Trainable Parameters Overhead Analysis

  • Purin adds Wout, contributing Cout additional parameters per convolutional layer and Nout per fully connected layer.The section calculates the relative overhead using the layers’ existing parameter counts and the added parameters.Source: C Additional Trainable Parameters… · p. 8
  • The final fully connected layer in all three models disables Wout and gstp and therefore adds no parameters.Figure 5 shows gstp curves for VGG11’s convolutional and fully connected layers, excluding its last fully connected layer.Source: Figure 5 · p. 9, C Additional Trainable Parameters… · p. 10
  • Purin’s additional trainable parameter overhead is below 0.2% for AlexNet, VGG11, and GoogLeNet, making a large increase in model capacity an unlikely explanation for accuracy improvements.The comparison assumes 224 × 224 images and 100 categories.Source: C Additional Trainable Parameters… · p. 10, Table 10 · p. 10

D Random Seed used in this Paper

Random seed 42 was used for dataset splits in the reported experiments.

D Random Seed used in this Paper

  • Random seed 42 was used to split CIFAR-100 and 102 Category Flower validation sets, and UCM and AID training, validation, and testing sets.The same seed was also used in supplementary experiments.Source: D Random Seed used in this Paper · p. 10
Loading 2609.31235v1…