Source-linked AI summary
Temporal-wise Attention Spiking Neural Networks for Event Streams Classification
Man Yao, Huanhuan Gao, Guangshe Zhao, Dingheng Wang, Yihan Lin, Zhaoxu Yang, Guoqi Li
TL;DR
Existing SNN processing of frame-aggregated event streams does not account for differing signal-to-noise ratios across sparse, non-uniform temporal frames. The paper introduces TA-SNNs with temporal attention, inference-time frame pruning, and RCS augmentation, achieving state-of-the-art results across three classification datasets while reducing input frames or latency.
Problem
Existing SNN models do not attach importance to serial frames with different signal-to-noise ratios after event streams are aggregated into frames.
Method
TA-SNNs learn temporal attention scores from event statistics, use thresholded binary scores to prune irrelevant inference frames, and employ RCS to augment event data.
Results
TA-SNNs and RCS achieve state-of-the-art accuracy on DVS128 Gesture, CIFAR10-DVS, and SHD: 98.61%, 72.00%, and 91.08%, respectively.
Takeaways & Limitations
IAP can obtain similar or better performance with only half the input frames, supporting lower latency and reduced computation for event-based processing.
Takeaways & Limitations
RCS requires inherently repetitive event-stream content and can reduce accuracy with long aggregation windows or fail on shorter SHD samples.
Abstract
from arXiv · showhide
How to effectively and efficiently deal with spatio-temporal event streams, where the events are generally sparse and non-uniform and have the microsecond temporal resolution, is of great value and has various real-life applications. Spiking neural network (SNN), as one of the brain-inspired event-triggered computing models, has the potential to extract effective spatio-temporal features from the event streams. However, when aggregating individual events into frames with a new higher temporal resolution, existing SNN models do not attach importance to that the serial frames have different signal-to-noise ratios since event streams are sparse and non-uniform. This situation interferes with the performance of existing SNNs. In this work, we propose a temporal-wise attention SNN (TA-SNN) model to learn frame-based representation for processing event streams. Concretely, we extend the attention concept to temporal-wise input to judge the significance of frames for the final decision at the training stage, and discard the irrelevant frames at the inference stage. We demonstrate that TA-SNN models improve the accuracy of event streams classification tasks. We also study the impact of multiple-scale temporal resolutions for frame-based representation. Our approach is tested on three different classification tasks: gesture recognition, image classification, and spoken digit recognition. We report the state-of-the-art results on these tasks, and get the essential improvement of accuracy (almost 19\%) for gesture recognition with only 60 ms.
1. Introduction
Event streams offer sparse, asynchronous, high-temporal-resolution sensing, but their temporal redundancy and varying frame quality complicate efficient processing. The paper proposes temporal-wise attention SNNs, plus IAP and RCS, to filter frames while preserving event-triggered computation.
- Motivation: DVS cameras encode pixel brightness changes as sparse, asynchronous events with microsecond temporal resolution, reducing resource use and motion blur.They are suited to low-latency, low-power visual tasks under varying illumination.
- Motivation: Temporal redundancy and irregular scene changes make event streams difficult for dense DNNs, while preprocessing dilutes their latency and power advantages.SNNs provide event-triggered computation that is naturally suited to event processing.
- Proposed approach: TA-SNNs extend attention to temporal inputs, learning frame significance during training and filtering irrelevant frames for the final decision.The TA module extracts event statistics at different times, generates attention scores, and weighs events accordingly.
- Proposed approach: Input attention pruning uses thresholded binary attention scores at inference, retaining event-encoded characteristics while discarding irrelevant frames.The method is designed to preserve or improve accuracy compared with full inputs.
- Proposed approach: Random consecutive slice augmentation selects consecutive frames from a random temporal starting point to make fuller use of sampled event data.The method is inspired by random temporal cropping in video recognition and overlap-based event-stream processing.
2. Related Works
Prior event-stream classification methods aggregate events into frames to obtain sufficient signal-to-noise ratios and compatibility with conventional vision pipelines. Related SNN and attention research provides the foundations for the paper’s frame-based temporal attention approach.
- Event Streams Classification: Frame-based representation aggregates event streams into groups, is easy to generate, and is compatible with traditional computer-vision frameworks and neuromorphic hardware.Larger temporal resolution generally provides higher signal-to-noise ratio.
- Spiking Neural Networks: Spiking neural networks transmit spike streams hierarchically, and event-triggered neurons can skip computation when no input spikes arrive.This characteristic supports power-saving information extraction from spikes.
- Attention Models: Attention mechanisms selectively focus on informative input components and have been used in sequence learning, translation, and action recognition.Prior work includes temporal-wise attention in recurrent networks and spatial-wise attention in SNNs.
3. Model Description
The model converts event streams into frame-based inputs, processes them with spiking neurons, and applies temporal-wise attention to estimate and filter frame relevance. Its attention module uses event statistics and neighboring-frame information to produce timestep scores for training and optional frame discarding during inference.
- 3.1. Frame-based Representation: Event streams are grouped into frames by aggregating β consecutive spike patterns, where β is the resolution factor.The frame timestep uses dt = dt′ × β and retains spatial and polarity dimensions.
- 3.1. Frame-based Representation: The frame aggregation function q(·) can use several strategies; this model accumulates events while preserving polarity information.The described alternatives include non-polarity, accumulate, and AND-logic aggregation.
- 3.2. Spiking Neural Network Models: LIF-SNNs combine spatial input from the previous layer with temporal input from the previous timestep, firing when membrane potential exceeds the threshold.Firing produces the next layer’s spike tensor, resets the membrane potential, and passes a decayed value temporally; LIAF replaces the Heaviside function with ReLU and uses analog values.
- 3.3. Temporal-wise Attention for SNNs: The TA module estimates each frame’s saliency from event-number statistics and neighboring-frame information through temporal squeeze and excitation operations.The squeeze step forms a statistical vector, while excitation uses a two-layer fully connected network to obtain correlations and attention scores across frames.
- 3.3. Temporal-wise Attention for SNNs: During training, the network uses the complete attention score vector; during inference, frames below dth can be discarded and retained-frame scores set to 1.The scores multiply the frame inputs to produce the new input tensor for the attention-enhanced SNN layer.
4. Experiments
Experiments evaluate TA-SNNs across gesture, event-based image, and spoken-digit classification, including TA placement, input pruning, temporal resolution, and comparisons with prior methods. The studies report improved accuracy, reduced computation, and strong performance across tasks.
- Experimental Setup: Experiments cover DVS128 Gesture, CIFAR10-DVS, and Spoken Heidelberg Digits using event-based gesture, image, and audio classification datasets.CIFAR10-DVS contains 10,000 samples with 300 ms duration; SHD contains 10,420 spoken-digit samples.
- Experimental Setup: RCS randomly selects consecutive frame sequences during training and aggregates ten crops with voting at test time.Event-stream latency is determined by dt × T, with overlapping crops used when fewer than ten non-overlapping crops fit.
- Gesture Recognition: TA inserted only at depth layers performs better than TA at the input layer or throughout the network in the position ablation.The depth-layer strategy is used for subsequent experiments because of its stability.
- Gesture Recognition: At a pruning proportion of 0.5, IAP maintains accuracy around 89%, whereas random input pruning decreases accuracy to around 78%.IAP discards low-attention-score frames during inference and can achieve similar or better performance with almost half the power.
- Gesture Recognition: For LIF, combining RCS and TA reaches 95.49% accuracy, while LIAF reports 98.61% without RCS under larger dt values.TA works across tested conditions; RCS helps most when dt is small and has weakened or negative effects at larger dt in some settings.
- Image Classification: On CIFAR10-DVS, temporal features are not dominant because frames generated by fixed image trajectories are similar, and both RCS and TA-SNNs improve accuracy.The reported setting uses T = 10 and dt = 10ms.
- Comparison with Prior Works: Across benchmark comparisons, TA-SNN models achieve state-of-the-art results on multiple event-based datasets with almost no parameter increase.The comparison includes CNNs, spike-based SNNs, and analog-based SNNs.
5. Discussion
TA-SNNs improve spatio-temporal feature extraction while retaining event-triggered processing, and attention-based pruning can reduce input frames with low latency. Performance depends on temporal resolution and sampling conditions.
- TA position: Depth-layer TA placement (S3) works better than input-only placement and consistently enhances spatio-temporal feature extraction over pure SNNs.The comparison is reported in the TA-position ablation study.
- Adaptability of TA module: 30–180 ms input-pruned inference can use only half the input frames while achieving similar or better performance than full-frame processing.The paper identifies this input attention pruning as a potential power-saving mechanism.
- Influence of RCS method: RCS requires inherently repetitive event-stream content, as found in DVS128 Gesture and CIFAR10-DVS.RCS randomly selects the training start time to make fuller use of sampled data.
- Influence of RCS method: 94.79% accuracy results when analog-based SNN accuracy falls from 98.61% to 94.79% under RCS with dt = 25ms.Long aggregation windows can destroy inherent periodicity; RCS also fails for shorter SHD samples that may become all zeros.
- TR Analysis: Short-term temporal resolution offers high-rate frames for low-latency tasks, while RCS and TA relax the associated low-SNR performance problem.IAP with short-term resolution can preserve or improve task accuracy in spike-based TA-SNNs.
6. Conclusions
TA-SNNs integrate temporal attention into SNNs for more effective event-stream processing while preserving event-triggered computation. Attention-based input pruning and RCS support the reported state-of-the-art results across three datasets.
- 6. Conclusions: TA-SNNs preserve SNN event-triggered processing while improving event-stream effectiveness and efficiency over pure LIF-SNNs.The model integrates temporal attention into SNNs.
- 6. Conclusions: Attention-score-based input pruning saves substantial computation without significant accuracy loss during inference.The pruning operation discards lower-attention frames.
- 6. Conclusions: The reported state-of-the-art accuracies are 98.61% on DVS128 Gesture, 72.00% on CIFAR10-DVS, and 91.08% on SHD.These results use TA-SNNs and RCS.
- 6. Conclusions: TA-SNNs retain sparse event-triggered characteristics that the paper identifies as significant for improving SNN accelerator performance.The accelerator implication is presented as a future-oriented significance.