PaperlayerView exact revision on arXiv

Source-linked AI summary

Attention Spiking Neural Networks

Man Yao, Guangshe Zhao, Hengyu Zhang, Yifan Hu, Lei Deng, Yonghong Tian, Bo Xu, Guoqi Li

arXiv:2209.13929v1cs.CV

TL;DR

SNNs provide energy-efficient, event-driven computation but retain a performance gap with ANNs and incur costs from multi-time-step processing. The paper introduces plug-and-play multi-dimensional attention that optimizes membrane potentials across temporal, channel, and spatial dimensions. Across event-based recognition and ImageNet-1K, MA improves sparsity, accuracy, and energy efficiency, including state-of-the-art SNN results.

  • Problem

    SNNs offer energy-efficient event-driven computation, but their performance gap with ANNs and multi-time-step costs limit broader deployment.

  • Method

    The paper proposes lightweight Multi-dimensional Attention that learns when, what, and where to focus and regulates spiking-neuron membrane potentials.

  • Results

    MA improves sparsity, performance, and energy efficiency across DVS128 Gesture/Gait and ImageNet-1K, reaching 77.08% top-1 accuracy on 4-step Res-SNN-104.

  • Takeaways & Limitations

    Attention can provide SNNs with a balance of effectiveness and efficiency while retaining their event-driven computation and supporting multiple tasks.

  • Takeaways & Limitations

    The energy analysis assumes 32-bit floating-point operations implemented in 45nm technology, and the paper uses rate-coding direct training.

Abstract

from arXiv · show

Benefiting from the event-driven and sparse spiking characteristics of the brain, spiking neural networks (SNNs) are becoming an energy-efficient alternative to artificial neural networks (ANNs). However, the performance gap between SNNs and ANNs has been a great hindrance to deploying SNNs ubiquitously for a long time. To leverage the full potential of SNNs, we study the effect of attention mechanisms in SNNs. We first present our idea of attention with a plug-and-play kit, termed the Multi-dimensional Attention (MA). Then, a new attention SNN architecture with end-to-end training called "MA-SNN" is proposed, which infers attention weights along the temporal, channel, as well as spatial dimensions separately or simultaneously. Based on the existing neuroscience theories, we exploit the attention weights to optimize membrane potentials, which in turn regulate the spiking response in a data-dependent way. At the cost of negligible additional parameters, MA facilitates vanilla SNNs to achieve sparser spiking activity, better performance, and energy efficiency concurrently. Experiments are conducted in event-based DVS128 Gesture/Gait action recognition and ImageNet-1k image classification. On Gesture/Gait, the spike counts are reduced by 84.9%/81.6%, and the task accuracy and energy efficiency are improved by 5.9%/4.7% and 3.4$\times$/3.2$\times$. On ImageNet-1K, we achieve top-1 accuracy of 75.92% and 77.08% on single/4-step Res-SNN-104, which are state-of-the-art results in SNNs. To our best knowledge, this is for the first time, that the SNN community achieves comparable or even better performance compared with its ANN counterpart in the large-scale dataset. Our work lights up SNN's potential as a general backbone to support various applications for SNNs, with a great balance between effectiveness and efficiency.

1 INTRODUCTION

SNNs offer event-driven, sparse computation for energy-efficient intelligence, but performance, latency, and energy gaps still limit deployment. The paper introduces attention that regulates membrane potentials across multiple dimensions to improve sparsity, performance, and efficiency together.

  • Motivation: SNNs use binary spike-based, event-driven communication that avoids computing zero activations and can consume less power than ANNs.This makes them attractive for resource- and battery-constrained platforms.
  • Motivation: SNNs still face a performance gap with ANNs, while multi-time-step computation increases training cost, inference latency, energy use, and memory overhead.These limitations motivate more effective and efficient SNN designs.
  • Research gap: Attention in SNNs remains underexplored, with the main adaptation challenges being retention of event-driven computation, support for diverse inputs, and binary spiking.The proposed approach adds attention as a lightweight auxiliary unit rather than replacing convolution or fully connected operators.
  • Approach: Attention can jointly improve effectiveness and efficiency by focusing on important features and inhibiting background-noise membrane potentials that would otherwise activate neurons.The authors use spiking-response visualization to connect proper focusing with effectiveness and increased sparsity with efficiency.
  • Approach: MA-SNN learns when, what, and where to attend through temporal, channel, and spatial attention dimensions while preserving the event-driven nature of SNNs.Attention weights optimize membrane potentials, regulating spiking responses in a data-dependent way.

2 PRELIMINARIES

This section reviews how deep SNNs are trained, how event-based vision represents brightness changes, and how attention has been used in CNNs. It positions direct training and attention-based representation enhancement as relevant foundations for the paper.

  • Training Methods of SNNs: ANN-to-SNN conversion approximates ANN activations with average firing rates but typically requires longer time steps to reduce rate-coding errors.Longer time steps increase latency and can restrict practical application.
  • Training Methods of SNNs: Direct SNN training uses surrogate gradients to enable backpropagation through non-smooth spiking and supports fewer time steps and temporal tasks.The paper uses rate-coding direct training to obtain large-scale SNNs.
  • Event-based Vision: Dynamic vision sensors encode pixel brightness changes as event streams containing time, location, and polarity with microsecond-level temporal resolution.Events are triggered only when intensity changes, reducing resource requirements and avoiding motion blur.
  • Attention in CNNs: Attention in CNNs is an architecture-design mechanism that increases representation power by focusing on important information alongside depth, width, and cardinality.The reviewed CNN literature includes channel attention and squeeze-and-excitation-style modules.

3 MULTI-DIMENSIONAL ATTENTION SPIKING NEURAL NETWORKS

The paper introduces multi-dimensional attention for SNNs, learning temporal, channel, and spatial weights to regulate membrane potentials and spiking responses. The design integrates lightweight attention with residual SNNs while preserving event-driven processing and addressing efficiency and deep-network limitations.

  • Multi-dimensional attention: The section develops MA-SNN, which learns temporal, channel, and spatial attention separately or jointly for SNNs.The attention dimensions correspond to when, what, and where information should be emphasized.
  • Conv-based SNN: The Conv-based LIF-SNN layer extracts spatial features with convolution, then integrates inputs over time and generates spikes through thresholding, reset, and leak dynamics.The membrane potential combines spatial features with temporal input, and its decay transmits state when no spike occurs.
  • Multi-dimensional attention: Attention weights refine membrane potentials in a data-dependent manner, regulating spiking responses while focusing on salient features and suppressing unnecessary ones.This design is motivated by attention's modulation of synaptic weights and neuronal spiking activity.
  • Temporal attention: Temporal attention aggregates spatial-channel information at each time step using average pooling, max pooling, and a shared MLP to produce temporal weights.The resulting weights are broadcast across channel and spatial dimensions.
  • Channel attention: Channel attention directly refines spiking-neuron membrane potentials and is intended to emphasize salient semantic attributes while suppressing minor features for improved SNN efficiency.Unlike CNN channel attention, this design operates on spatio-temporal fusion information in SNNs.
  • Attention residual learning: MA integrates into existing residual SNN architectures by applying attention to basic Res-SNN outputs, with Att-Res-SNN-1 recommended for attention residual learning.The residual design aims to preserve event-driven processing while avoiding degradation in deep SNNs.

4 ANALYSIS OF ENERGY CONSUMPTION

The paper analyzes attention-SNN energy by accounting for added MAC operations against reductions in spike-driven AC operations. It defines activity-based indicators and relative energy efficiency to quantify this trade-off.

  • Energy Shift of Attention SNNs: MA-SNN regulates membrane potentials to reduce spiking activity, so its energy increase comes from attention-related MAC operations while AC operations decrease.The reduction applies to convolutional and fully connected layers.
  • Energy Cost of Vanilla SNNs: Vanilla-SNN energy combines MAC operations in the analog-input encoder with AC operations in subsequent convolutional and fully connected layers.The energy model assumes 32-bit floating-point operations in 45 nm technology, with EMAC = 4.6 pJ and EAC = 0.9 pJ.
  • Additional Model and Computational Complexity: Attention adds computation through attention-weight generation and membrane-potential refinement, while its parameters remain a small fraction of total network capacity.Additional parameters come from two fully connected layers for temporal/channel attention or one convolutional layer for spatial attention.
  • Energy Shift of Attention SNNs: Relative energy efficiency compares attention-SNN energy with vanilla-SNN energy, using the baseline model as 1×; higher rEE indicates greater energy efficiency.The energy shift is estimated from additional MAC operations and decreased AC operations.
  • Network Average Spiking Activity Rate: NASAR averages the spiking activity rate across time steps, and spike count equals NASAR · neuron number · T.Lower spike counts indicate better energy efficiency because each spike corresponds to accumulated AC operations whose amount depends on network architecture.

5 EXPERIMENTS

Experiments evaluate MA-SNN across event-based Gesture/Gait recognition and ImageNet-1K classification, using matched vanilla baselines and attention counterparts. Across these settings, attention improves accuracy, reduces spiking activity or energy cost, and supports efficient single- and multi-step SNNs.

  • Experimental scope: Experiments cover DVS128 Gesture/Gait action recognition and ImageNet-1K static image classification.The study evaluates MA-SNN across multiple datasets and task types.
  • Experimental setup: Matched vanilla and attention models use the same network structures, hyperparameters, learning methods, and evaluation setup.MA is added as a plug-and-play module to re-implemented baselines.
  • Event-based recognition: 5.9% and 4.7% accuracy gains are obtained on Gesture and Gait with the three-layer attention model.The gains are also consistent for the five-layer vanilla model comparison.
  • ImageNet-1K classification: 75.92% single-step and 77.08% four-step top-1 accuracy are achieved by CSA-Res-SNN-104 on ImageNet-1K.The four-step result is reported as state of the art in SNNs and exceeds the same-architecture ANN’s 76.87% accuracy.
  • ImageNet-1K efficiency: Attention reduces ImageNet-1K inference energy while improving accuracy, including 31.8× compute energy efficiency for single-step CSA-Res-SNN-104 versus Res-ANN-104.For CSA-Res-SNN-18, NASAR falls from 0.224 to 0.148 while accuracy rises from 61.70% to 63.97%.
  • Training efficiency: Single-step large-scale attention SNNs provide comparable or better accuracy than multi-step baselines while achieving 6.4× training acceleration and requiring less hardware.The results motivate focusing on single-step simulation before extending it to multi-step SNNs.

6 ABLATION STUDY

The ablations examine where to place attention, which attention modules to use, and how to combine temporal, channel, and spatial dimensions in SNNs. Optimizing membrane potentials across these design choices consistently improves performance while reducing spiking activity, with three-dimensional attention providing the strongest reported combination.

  • 6.1 Attention Locations in Plain SNNs: Conv-POST improves temporal-attention performance over Conv-PRE with similar NASAR, while Activate-PRE gives channel attention the best performance and lowest NASAR.Activate-PRE channel attention directly optimizes the spatial-temporal fused membrane potential.
  • 6.1 Attention Locations in Plain SNNs: Individual spatial attention improves performance and lowers NASAR without strong sensitivity to location, leading the authors to use serial channel-then-spatial attention at Activate-PRE.This setting follows the common practice described for attention CNNs.
  • 6.1 Attention Locations in Plain SNNs: Attention location affects the balance of performance and NASAR, motivating separate placements for temporal, channel, and spatial attention in plain SNNs.The recommended MA-SNN locations are summarized in Fig. 4 after comparing Conv-PRE, Conv-POST, and Activate-PRE alternatives.
  • 6.2 Different Attention Variants: CBAM, ECA, and SimAM all improve performance while producing sparser spiking activity, with CBAM giving the highest gains, ECA the smallest ∆MAC, and SimAM no added parameters.The NASAR values are close across the three modules, so the module can be selected according to scenario requirements.
  • 6.3 Combinations of Attention Dimension: +5.9 accuracy gain and 3.4× energy efficiency are achieved by three-dimensional TCSA-SNN over vanilla SNN in the Gesture ablation.TCSA-SNN exceeds all two-dimensional and single-dimensional attention SNNs, while attention-dimension combinations remain broadly robust.

7 UNDERSTANDING AND VISUALIZING ATTENTION

The analysis explains attention SNN behavior through gradient stability and spiking-response visualizations. Attention can support stable residual networks, focus activity on relevant features, suppress redundant channels, and produce sparser spiking responses without sacrificing performance.

  • 7.1 Gradient Evolvement in Att-Res-SNNs: Att-Res-SNN-1 and Att-Res-SNN-2 can achieve block dynamical isometry, allowing attention residual networks to be as stable as a shallower network.The theorem assumes designs with L sequential blocks and connects their stability to gradient norm equality.
  • 7.1 Gradient Evolvement in Att-Res-SNNs: The analysis decomposes attention-residual Jacobians across channel, spatial, and channel-spatial blocks to establish conditions for stable gradient evolution.The proof evaluates serial channel-spatial attention and the two proposed attention residual designs.
  • 7.4 Spiking Response of Attention SNNs: 0.51 versus 0.24 is the highest Conv-3 spiking response for vanilla SNN and TCA-SNN on Gesture, respectively.The decrease is consistent with the NASAR comparisons and is attributed to attention-driven membrane-potential optimization.
  • 7.3 Average Spiking Response Visualization: ASRV averages spiking tensors across validation samples so each feature-map pixel represents one neuron’s spiking activity rate.The method compares vanilla, temporal-, channel-, and temporal-channel-attention SNNs across selected time steps and channels.
  • 7.4 Spiking Response of Attention SNNs: Attention visualizations show focus on target features and suppression of redundant background channels, including channels whose neurons have zero firing rate.In CSA-Res-SNN, 17, 13, and 21 dark-blue channels appear in layers 2, 4, and 6, respectively.
  • 7.4 Spiking Response of Attention SNNs: Attention produces richer object features in the final convolutional layer of Att-Res-SNN-34 than in its vanilla counterpart.The authors connect the richer last-layer features to the importance of final convolutional features for correct classification.

8 CONCLUSION

The paper concludes that lightweight multi-dimensional attention can improve SNN performance and energy efficiency while preserving event-driven operation. Across event-based and ImageNet tasks, attention produces sparser activity and state-of-the-art results, supported by spiking-response visualizations.

  • 8 CONCLUSION: MA is a plug-and-play attention module for convolution-based SNNs that learns when, what, and where to focus or suppress through temporal, channel, and spatial pathways.The module optimizes spiking-neuron membrane potentials, which reduces spiking activity and improves performance.
  • 8 CONCLUSION: Experiments on DVS128 Gesture/Gait and ImageNet-1K report state-of-the-art performance and significant energy efficiency across multiple datasets and tasks.The conclusion also states that attention-induced sparser activity is examined through visualizations comparing vanilla and attention SNNs.
Loading 2209.13929v1…