Comparing SNNs and RNNs on Neuromorphic Vision Datasets: Similarities and Differences

Weihua He; YuJie Wu; Lei Deng; Guoqi Li; Haoyu Wang; Yang Tian; Wei Ding; Wenhui Wang; Yuan Xie

Source-linked AI summary

Comparing SNNs and RNNs on Neuromorphic Vision Datasets: Similarities and Differences

Weihua He, YuJie Wu, Lei Deng, Guoqi Li, Haoyu Wang, Yang Tian, Wei Ding, Wenhui Wang, Yuan Xie

arXiv:2005.02183v1May 2, 2020cs.CVcs.NEeess.IV

TL;DR

The paper addresses the unclear relative performance of SNNs and RNNs on neuromorphic data by systematically comparing them on vision datasets. It standardizes training and architecture choices, then evaluates converted and captured datasets across temporal resolutions. The reported conclusions favor SNNs over common RNNs in accuracy and compactness, while rate-coding-inspired training can make LSTM comparable or slightly better.

Problem
The study asks how SNNs and RNNs compare on neuromorphic data, where RNN evaluation remains limited despite their ability to model temporal dynamics.
Method
It systematically compares feedforward SNNs, vanilla RNNs, and LSTM using unified datasets, temporal resolutions, BPTT-based learning, rate-coding-inspired losses, architectures, and training hyper-parameters.
Results
SNNs usually achieve better accuracy than common RNNs, while the rate-coding-inspired loss can make LSTM comparable to or slightly better than SNNs.
Takeaways & Limitations
The findings support selecting models according to workload: SNNs for small temporal resolution or compact models, and rate-coding-trained LSTM for large temporal resolution.

Abstract

from arXiv · show

Neuromorphic data, recording frameless spike events, have attracted considerable attention for the spatiotemporal information components and the event-driven processing fashion. Spiking neural networks (SNNs) represent a family of event-driven models with spatiotemporal dynamics for neuromorphic computing, which are widely benchmarked on neuromorphic data. Interestingly, researchers in the machine learning community can argue that recurrent (artificial) neural networks (RNNs) also have the capability to extract spatiotemporal features although they are not event-driven. Thus, the question of "what will happen if we benchmark these two kinds of models together on neuromorphic data" comes out but remains unclear. In this work, we make a systematic study to compare SNNs and RNNs on neuromorphic data, taking the vision datasets as a case study. First, we identify the similarities and differences between SNNs and RNNs (including the vanilla RNNs and LSTM) from the modeling and learning perspectives. To improve comparability and fairness, we unify the supervised learning algorithm based on backpropagation through time (BPTT), the loss function exploiting the outputs at all timesteps, the network structure with stacked fully-connected or convolutional layers, and the hyper-parameters during training. Especially, given the mainstream loss function used in RNNs, we modify it inspired by the rate coding scheme to approach that of SNNs. Furthermore, we tune the temporal resolution of datasets to test model robustness and generalization. At last, a series of contrast experiments are conducted on two types of neuromorphic datasets: DVS-converted (N-MNIST) and DVS-captured (DVS Gesture).

I. INTRODUCTION

Neuromorphic vision datasets encode dynamic pixel changes as spike events, combining spatial and temporal information in an event-driven format. The paper compares SNNs and RNNs on converted and naturally captured datasets under unified evaluation settings.

Dataset Motivation: Neuromorphic vision datasets record pixel-intensity changes as binary spike events, preserving interacting spatial and temporal information.They support applications including visual recognition, motion segmentation, tracking control, and robotics.
SNNs and RNNs: SNNs use membrane-potential dynamics and threshold-triggered binary spikes, whereas RNNs can process spike events as ordinary binary values while retaining previous states.This creates a direct comparison between event-driven SNNs and non-event-driven recurrent ANNs.
Study Design: The study systematically compares SNNs and RNNs by unifying BPTT, rate-coding-inspired losses, network structures, and training hyper-parameters.It also varies temporal resolution to test robustness and generalization.
Study Design: Experiments cover DVS-converted N-MNIST and DVS-captured DVS Gesture, spanning recognition accuracy, feature extraction, temporal resolution, generalization, complexity, and parameter volume.The experiments are intended to inform model selection and construction.
Dataset Types: DVS-converted data derive spikes from repeated camera movement over static images, whereas DVS-captured data arise from natural motion with essential spatial and temporal structure.N-MNIST illustrates the converted setting; DVS Gesture illustrates the captured setting.

B. Spiking Neural Networks

The paper uses LIF-based feedforward SNNs and compares them with vanilla RNNs and LSTM models that represent temporal information through continuous recurrent states or gated memory.

Spiking Neural Networks: The SNNs use leaky integrate-and-fire neurons connected in feedforward networks with stacked fully connected or convolutional layers.LIF is selected because of its lower complexity and widespread practical use.
Spiking Neural Networks: A LIF neuron maintains continuous membrane potential and binary output activity, firing and resetting when its potential crosses a threshold.Otherwise, the membrane potential leaks over time.
Recurrent Neural Networks: Vanilla RNNs introduce temporal dynamics through recurrent connections and maintain one continuous hidden-state variable per neuron.Their recurrent computation uses inter-layer and intra-layer weights with an activation function such as tanh.
Recurrent Neural Networks: LSTM extends vanilla RNNs with forget, input, and output gates plus cellular and hidden states to improve long-term temporal dependence.The gates are designed to alleviate gradient vanishing.

III. METHODOLOGY

The methodology frames feedforward SNNs, vanilla RNNs, and LSTM as models with comparable spatiotemporal propagation while preserving differences in neuron states, recurrence, and gating. It uses supervised spatiotemporal backpropagation to make learning comparisons fair.

Scope: The comparison is explicitly between feedforward SNNs and recurrent ANNs, represented by vanilla RNNs and LSTM.The study analyzes their modeling and learning similarities and differences.
Neuron and Gradient Models: The LIF model incorporates membrane-potential leakage, threshold firing, and reset, while surrogate derivatives are used to support backpropagation through the non-differentiable firing function.The gradient-width parameter controls the approximation used during backward propagation.
Spatiotemporal Dynamics: SNNs, vanilla RNNs, and LSTM all propagate information across spatial layers and timesteps, producing broadly similar forward and backward spatiotemporal paths.LSTM includes additional internal gate and cell states, while SNN membrane potential directly affects the next timestep.
Learning: The study adopts a BPTT-inspired supervised spatiotemporal backpropagation algorithm for SNNs to align learning with RNN training.The approach is selected because RNNs are commonly trained with gradient-descent-based supervised learning.

B. Rethinking the Differences

SNNs, vanilla RNNs, and LSTM share spatiotemporal information propagation but differ in recurrent connections, neuron structure, activation functions, and loss formulations.

Connection Pattern: SNNs use self-recurrence with restricted leakage-determined weights, whereas RNNs use trainable cross-neuron recurrent weights.Figure 4 depicts connection patterns, while Figure 5 presents recurrent-weight distributions from practical models.
Neuron Model: SNN membrane-potential paths resemble LSTM forget-gate behavior, but reset bounds the membrane potential unlike LSTM’s unbounded cellular state.Vanilla RNNs have no gates, whereas LSTM uses multiple gates and cellular state.
Neuron Model: SNNs use binary step-like firing outputs, while vanilla RNNs and LSTM use continuous tanh and sigmoid activations.Figure 7 contrasts firing in SNNs with tanh and sigmoid functions in RNNs.
Loss Function: SNN loss combines spike-rate coding with MSE and uses outputs from all timesteps, whereas mainstream RNN loss uses the final hidden state.The RNN formulation also includes a trainable output weight matrix.

C. Comparison Methodology

The evaluation makes SNN–RNN comparisons possible by recognizing their shared spatiotemporal dynamics and unifying key experimental factors for fairness.

Comparison Methodology: The study unifies datasets, temporal resolution, learning algorithm, loss function, network structure, and training hyperparameters to ensure comparability and fairness.This standardization follows the models’ shared spatiotemporal dynamics while addressing their differences.

1) Dataset Selection and Temporal Resolution Tuning:

The study evaluates models on N-MNIST and DVS Gesture while collapsing successive event slices to tune temporal resolution and control simulation length.

Dataset Selection: The benchmark uses N-MNIST, a DVS-converted dataset, and DVS Gesture, a DVS-captured dataset.For RNNs, spike events are treated as binary values {0, 1}.
Temporal Resolution Tuning: Temporal collapse combines multiple successive event slices into one slice by placing a spike at a pixel when any slice in the window contains one.The collapsed sequence has fewer timesteps, with the new sequence indexed by the temporal-resolution factor.
Temporal Resolution Tuning: Increasing temporal resolution raises the new slices’ spike rate and can capture more temporal dependence when the simulation timestep count remains fixed.The same fixed number of simulation steps then covers a longer actual simulation time.

2) Learning Algorithm and Loss Function Design:

The comparison uses supervised BPTT for both model families and provides RNN loss configurations that better align with the SNN rate-coding objective.

Learning Algorithm: SNNs use a BPTT-inspired algorithm with spatiotemporal gradient propagation to match the supervised BPTT training used for RNNs.This choice is intended to make the comparison fair.
Loss Function: The SNN rate-coding MSE loss receives gradient feedback at every timestep, which can alleviate vanishing gradients to some extent.It uses outputs across the simulation sequence rather than only a terminal output.
Loss Function: RNNs are evaluated with both the mainstream loss and a modified rate-coding-inspired loss to reduce differences from the SNN objective.The modified configuration is introduced because existing all-timestep RNN losses remain slightly different from the SNN formulation.

3) Network Structure and Hyper-parameter Setting:

The study matches SNNs and RNNs with common architectures, training settings, and evaluation procedures to make their neuromorphic-data comparison fair. Experiments then examine accuracy, feature extraction, membrane mechanisms, and convergence across N-MNIST and DVS Gesture.

Network structure: CNN structures are implemented for SNNs, vanilla RNNs, and LSTM, while N-MNIST uses MLPs and DVS Gesture uses both MLP and CNN structures.
Hyper-parameters: Training hyper-parameters are unified across models except for SNN-specific settings, including epochs, timesteps, batch size, and learning rate.
Comparison protocol: The comparison unifies datasets, temporal resolution, BPTT, loss function, network structure, and hyper-parameters to improve fairness.
Accuracy: On N-MNIST and DVS Gesture, rate-coding-inspired loss enables RNNs to achieve comparable or better accuracy than SNNs, although vanilla RNNs remain weaker on DVS Gesture.
Training dynamics: SNNs and LSTM converge faster than vanilla RNNs, while the modified loss shifts RNN training curves upward on N-MNIST.
Feature extraction: On DVS Gesture, SNN and LSTM feature maps integrate current inputs with previous traces, whereas vanilla RNN maps remain closer to individual input slices.
SNN mechanisms: Removing membrane-potential leakage and reset degrades SNN accuracy, and their joint effect is larger than either mechanism alone.

C. Temporal Resolution Analysis

Temporal resolution affects both accuracy and model preference. Larger resolutions generally improve accuracy, while SNNs show an advantage on sparse, DVS-captured data at small resolutions and retain stronger cross-resolution generalization.

Accuracy versus resolution: Accuracy improves as temporal resolution increases across the evaluated neuromorphic datasets.Larger dt produces denser slices and includes more of the moving object within the same number of simulation timesteps.
Dataset dependence: SNNs achieve significant accuracy superiority on DVS Gesture when dt ≤10ms, whereas their advantage is reduced on DVS-converted N-MNIST.
Generalization test: The generalization test trains models at dt = 3ms and evaluates them at dt = 3ms, 2ms, and 1ms while keeping T × dt = 45ms.
Generalization mechanism: SNN robustness is attributed to event-driven sparse-feature processing and restricted self-neuron recurrence, while trainable cross-neuron recurrence causes generalization to degrade like RNNs.
Generalization mechanism: Adaptive leakage factors only slightly improve SNN robustness under changing temporal resolution.

D. Loss Function & Temporal Contrast

The study analyzes temporal contrast through cross-entropy between event slices to explain when the rate-coding-inspired loss benefits RNNs. DVS-captured data have stronger temporal contrast, but excessive contrast can undermine the loss’s effectiveness.

Temporal contrast: Temporal contrast is defined as the cross-entropy between slices separated across timesteps, using S[tx, tx+k] and S[ty, ty+k].
Temporal contrast: Binary slice values require an epsilon-stabilized logarithm to avoid zero or negative-infinity results during cross-entropy calculation.
Dataset comparison: DVS-captured DVS Gesture has much larger temporal contrast than DVS-converted N-MNIST, indicating more temporal components in the captured data.
Limitation: When temporal contrast is too large, the rate-coding-inspired loss may become less effective because gradients from different timesteps diverge.

E. Number of Parameters and Operations

The comparison evaluates computational complexity and parameter volume as practical costs of SNNs and RNNs. SNNs use fewer parameters while maintaining competitive recognition accuracy through membrane-potential dynamics and restricted recurrence.

Operations: Computational complexity is compared by counting matrix operations in forward and backward propagation for a layer with M neurons and N downstream neurons.
Parameter volume: SNNs use fewer parameters than recurrent baselines on DVS Gesture, occupying about 80% of vanilla RNN parameters and 20% of LSTM parameters.
Accuracy-cost trade-off: Despite fewer operations and parameters, SNNs achieve comparable accuracy to LSTM at large temporal resolutions and can exceed it at small temporal resolutions.
Accuracy-cost trade-off: Self-neuron recurrence and restricted recurrent weights make SNNs more lightweight and robust than the compared recurrent models.

F. Discussion on Audio Data

An additional Spoken-Digits experiment extends the comparison beyond vision data. On this audio dataset, SNNs perform best and generalize better when testing temporal resolution varies.

Audio experiment: The study extends its scope to audio by testing SNNs and RNNs on the Spoken-Digits dataset.The network is Input-512FC-10, with T = 75 during training and varied during testing.
Audio experiment: SNNs achieve the best performance on Spoken-Digits, while vanilla RNNs perform the worst.
Audio experiment: SNNs also show better generalization on Spoken-Digits when the testing simulation timesteps vary.
Broader comparison: Across the broader benchmark, rate-coding-inspired loss improves RNN accuracy, especially for LSTM, making it comparable to or slightly better than SNNs.
Broader comparison: The study reports that SNNs are especially advantageous for sparse spike events at small temporal resolutions, while LSTM benefits from longer temporal contexts.