Source-linked AI summary
First-spike based visual categorization using reward-modulated STDP
Milad Mozafari, Saeed Reza Kheradpisheh, Timothée Masquelier, Abbas Nowzari-Dalini, Mohammad Ganjtabesh
TL;DR
The paper addresses whether reinforcement learning can train an SNN for natural-image object recognition without an external classifier. It uses a feedforward, temporally coded SNN with reward-modulated STDP and first-spike classification, and reports stronger task-specific feature extraction than STDP across three datasets. The method also remains constrained by a shallow trainable architecture and limited robustness to large object variation.
Problem
Unsupervised STDP favors frequent features and can miss rare but diagnostic features important for decision-making.
Method
A hierarchical SNN uses R-STDP to reinforce or suppress class-specific neurons according to whether their earliest spike gives the correct category decision.
Results
R-STDP strongly outperformed STDP and a same-structure CNN on Caltech Face/Motorbike, ETH-80, and NORB, with accuracies of 98.9%, 89.5%, and 88.4%, respectively.
Takeaways & Limitations
The approach extracts task-specific diagnostic features while performing feature extraction and categorization through spikes without an external classifier.
Takeaways & Limitations
The method uses only one trainable layer and cannot resist high object-variation rates without increasingly many neurons.
Abstract
from arXiv · showhide
Reinforcement learning (RL) has recently regained popularity, with major achievements such as beating the European game of Go champion. Here, for the first time, we show that RL can be used efficiently to train a spiking neural network (SNN) to perform object recognition in natural images without using an external classifier. We used a feedforward convolutional SNN and a temporal coding scheme where the most strongly activated neurons fire first, while less activated ones fire later, or not at all. In the highest layers, each neuron was assigned to an object category, and it was assumed that the stimulus category was the category of the first neuron to fire. If this assumption was correct, the neuron was rewarded, i.e. spike-timing-dependent plasticity (STDP) was applied, which reinforced the neuron's selectivity. Otherwise, anti-STDP was applied, which encouraged the neuron to learn something else. As demonstrated on various image datasets (Caltech, ETH-80, and NORB), this reward modulated STDP (R-STDP) approach extracted particularly discriminative visual features, whereas classic unsupervised STDP extracts any feature that consistently repeats. As a result, R-STDP outperformed STDP on these datasets. Furthermore, R-STDP is suitable for online learning, and can adapt to drastic changes such as label permutations. Finally, it is worth mentioning that both feature extraction and classification were done with spikes, using at most one spike per neuron. Thus the network is hardware friendly and energy efficient.
1 Introduction
The paper introduces R-STDP as a reinforcement-learning approach for training spiking neural networks to recognize natural-image objects without an external classifier. Across three benchmarks, the method targets task-specific diagnostic features and reports high recognition accuracies.
- Unsupervised STDP detects frequent input patterns but can miss rare diagnostic features needed for decision-making.
- R-STDP modulates or reverses STDP weight changes using reward or punishment signals tied to network outcomes.
- The proposed hierarchical SNN assigns class-specific neurons to fire early for target stimuli, enabling classification from first-spike latencies without an external classifier.
- R-STDP uses reward feedback to find task-specific discriminative features, while toy experiments showed reduced computational-resource requirements relative to STDP.
- 98.9% recognition accuracy was reached on Caltech face/motorbike, 89.5% on ETH-80, and 88.4% on NORB.
2 Materials and Methods
The network is a four-layer feedforward SNN that converts oriented-image features into spike latencies, learns complex features with R-STDP, and classifies images by the earliest category-associated spike.
- The four-layer architecture alternates simple and complex layers: S1, C1, S2, and C2.
- S1 detects oriented edges and converts their saliency into spike latencies, while C1 applies local pooling for some position invariance.
- S2 integrate-and-fire neurons combine C1 inputs into complex features and use pre-spike time, post-spike time, and reward or punishment for plasticity.
- C2 neurons are assigned to categories, and the category whose neuron fires first becomes the network decision and generates reward or punishment.
2.3 Layer C1
C1 performs local pooling over S1 spike maps, propagating the earliest local input while using lateral inhibition to reduce redundancy and favor salient information.
- Each C1 neuron pools a local S1 window and emits a spike immediately after receiving its earliest input spike.
- The C1 firing time is computed as the minimum firing time among its presynaptic S1 neurons.
- Lateral inhibition suppresses same-position responses in other orientation grids and delays nearby neurons within the same grid.
- S2 receives pooled C1 information through plastic synapses and updates membrane potential from weighted presynaptic spikes.
- S2 synaptic weights are updated from pre- and postsynaptic spike order together with reward or punishment feedback from the decision layer.
2.5 Layer C2
C2 is the decision layer: category-associated neurons propagate the earliest S2 spike, and the resulting category determines reward or punishment.
- Each C2 neuron receives S2 activity from one neuronal grid and propagates only the first spike arriving from that grid.
- C2 neurons are divided into category groups, and the group with the earliest spike determines the input-image category.
- A correct decision produces reward, whereas an incorrect decision produces punishment for synaptic plasticity.
- If no C2 neuron fires, no reward or punishment is generated and no weight change is applied.
2.6 Reward-Modulated STDP (R-STDP)
R-STDP updates the earliest-firing S2 neuron's synaptic weights according to whether its category decision is correct, using reward or punishment to reinforce or reverse STDP.
- The earliest-firing S2 neuron wins the competition, determines the network’s decision, and is the only neuron whose synaptic weights are updated.
- A reward or punishment signal modulates the magnitude of presynaptic weight changes according to the correctness of the network’s decision.
- The reward and punishment rules apply when presynaptic cells spike or remain silent, with weight-change magnitude scaled by parameters a+ and a−.
- Under punishment, anti-STDP reverses STDP polarity by swapping long-term depression with long-term potentiation.
- The learning rule uses an infinite time window rather than the exact difference between pre- and postsynaptic spike times.
2.7 Overfitting Avoidance
The network addresses reinforcement-learning overfitting and dead or overly selective neurons by balancing update effects and encouraging broader neuron participation during training.
- Reinforcement learning can trap the network in local optima or overfit to maximizing reward on training examples.
- Early training produces many punishments, which can rapidly weaken synapses and create dead or highly selective neurons covering few inputs.
- An adjustment factor balances the impact of correctly and incorrectly classified samples across training iterations.
- The counts of correctly and incorrectly classified samples may not sum to all training samples because some inputs activate no S2 neurons.
- Finding only a few sufficient features can cause severe overfitting and leave some neurons untrained, reducing test-set hit rate.
- Training-only dropout temporarily turns off C2 neurons, increasing participation in finding discriminative features and decreasing blind firings.
2.8 Classification
Classification uses the earliest spike in the final layer, with category assignments for C2 neurons, so decisions rely on spiking activity without an external classifier.
- The earliest spike in layer C2 is the sole information used for the final decision, eliminating the need for an external classifier.
- For m categories, layer S2 contains n = k × m neuronal grids, with k features associated with each category.
- During training, the network compares each decision with the stimulus label and generates reward or punishment when it matches or mismatches.
- STDP and R-STDP models are evaluated using K-nearest neighbors and support vector machine classifiers on extracted feature vectors.
- The first-spike feature vector is binary, with a single one marking the neuronal grid whose spike occurs earliest.
- Spike-count and potential vectors instead represent total spikes per grid and maximum membrane potential per grid, respectively.
3 Results
The experiments show that R-STDP learns task-specific discriminative features more efficiently than STDP, supports flexible adaptation, and improves recognition across multiple image benchmarks. Performance gains were observed in toy temporal discrimination, changing-target tasks, and natural-image categorization.
- 3.1 R-STDP Increases Computational Efficiency: R-STDP identified temporal features that distinguished similar inputs in 98% of trials, whereas standard STDP never found them under the same settings.With larger receptive fields and higher thresholds, STDP found the discriminative features in 80% of trials.
- 3.2 Plastic Neurons: R-STDP enabled neurons to unlearn previous target features and relearn swapped targets after punishment in a changing environment.After the target swap, punishment weakened prior selectivity, and neurons converged to the new targets after iteration 300.
- 3.3.2 Reinforced Selectivity: On Caltech face/motorbike recognition, R-STDP achieved 98.9 ± 0.4% accuracy across 30 runs, compared with 97.2% for the best same-structure STDP network.Training and testing sets and random initial weights varied across repetitions.
- 3.3.3 Performance: On ETH-80, R-STDP reached 89.5 ± 1.9% performance, and reinforcement learning reduced confusion while producing a more balanced error distribution.The comparison used the same network structure, with STDP evaluated using linear-kernel SVMs and KNNs.
- 3.3.3 Performance: On NORB, R-STDP achieved 88.4±0.5% testing performance, whereas STDP achieved at most 66%.Both methods struggled particularly with distinguishing four-leg animals from humans and cars from trucks.
- 3.3.4 Overfitting Problem: Adaptive learning rates improved test performance, while the suitable dropout probability depended on dataset and network complexity.NORB required more dropout than ETH-80 because its more complex samples were more prone to overfitting.
4 Discussion
The proposed hierarchical SNN uses R-STDP and first-spike latencies to perform visual feature extraction and categorization without an external classifier. Across natural-image benchmarks, R-STDP found more task-specific diagnostic features and outperformed comparable STDP-based approaches, while remaining limited relative to state-of-the-art deep learning.
- Method: R-STDP trains class-specific neurons to identify image categories from first-spike latencies in a hierarchical SNN.Neurons are assigned to classes and reinforced to fire early for matching stimuli.
- Resource efficiency: R-STDP solved a hand-designed discrimination task with fewer neurons, synapses, and lower thresholds than STDP.STDP required more resources because it tends to learn statistically frequent rather than diagnostic features.
- Visual categorization: R-STDP outperformed STDP and a same-structure CNN on Caltech Face/Motorbike, ETH-80, and NORB using first-spike decisions alone.The comparison included networks evaluated with powerful classifiers, whereas the proposed approach classified from spike timing.
- Implementation: The network permits one spike per neuron per image, reducing energy consumption while keeping classification based on first-spike latencies.The authors describe this combination as biologically plausible and hardware-friendly.
- Limitations: The method remains less competitive than state-of-the-art deep learning and is constrained by its single trainable layer and large receptive fields.The authors identify deeper architectures and multilayer plasticity as future directions.
- Future directions: R-STDP may also support semantic associations that group diagnostic features into broader categories such as animal or face.This is presented as a future research direction rather than an evaluated result.
“Supplementary Materials” First-spike based visual categorization using
The supplementary materials identify the paper’s authors and their affiliations across the University of Tehran, IPM, and CNRS Université Toulouse 3.
- Authors: The paper lists Milad Mozafari, Saeed Reza Kheradpisheh, Timothée Masquelier, Abbas Nowzari-Dalini, and Mohammad Ganjtabesh as authors.
- Affiliations: The authors are affiliated with the University of Tehran’s Department of Computer Science and School of Biological Sciences at IPM.
- Affiliations: Timothée Masquelier is affiliated with CerCo UMR 5549 at CNRS Université Toulouse 3 in France.
1 Datasets
The study evaluates the network on three object-recognition benchmarks, using training images for per-image synaptic plasticity and testing images for evaluation. The datasets span two-class, eight-class, and five-class recognition problems, including viewpoint and lighting variation.
- Evaluation protocol: Training images drive per-image synaptic plasticity, while testing images evaluate the network’s ability to extract general features.Each training iteration processes the training set before evaluation on the testing set.
- Caltech face/motorbike: Caltech face/motorbike contains faces and motorbikes, with 200 training images and 235 testing images randomly selected per category.The benchmark is used as a two-class recognition task.
- ETH-80: ETH-80 contains eight object categories, 10 instances per category, and 41 viewpoints per instance for assessing view-invariant recognition.Leave-one-out cross-validation separates object instances across training and testing.
- NORB: NORB contains five object classes imaged across viewpoints and lighting conditions, with a subsampled protocol producing 1,800 training and 1,800 testing images.The subsampling fixes one camera, one lighting level, four elevations, and all 18 azimuths.
2 Results
R-STDP extracts category-specific features and yields more balanced discrimination than STDP, while dropout improves feature coverage and reduces blind firings. Comparisons with CNN and VGG16 show that performance depends on architecture, training strategy, and overfitting control.
- Feature Extraction: R-STDP distinguishes similar objects by extracting category-specific features, such as tomato-head structure for separating apples from tomatoes.
- Confusion Matrix: R-STDP produces more symmetric confusion matrices than STDP, indicating more balanced discrimination across similar categories.The asymmetry measure is closer to 0 for more symmetric matrices; reported values are 0.16 for R-STDP and 0.85 for STDP.
- Dropout: Dropout improves performance by increasing the chance of finding discriminative features and decreasing blind firings.Green bars count correct earliest spikes, whereas red bars count false alarms.
- Comparison with Convolutional Neural Network: The CNN comparison used grayscale inputs, convolutional and max-pooling layers, followed by dense ReLU and Softmax layers.
- Comparison with Deep Convolutional Neural Network: VGG16 always outperformed the proposed network when fine-tuned, while training from scratch caused serious overfitting on NORB.The authors note that VGG16 performance depended on fine-tuning strategy and parameter values.