Source-linked AI summary
Event-based Video Reconstruction via Potential-assisted Spiking Neural Network
Lin Zhu, Xiao Wang, Yi Chang, Jianing Li, Tiejun Huang, Yonghong Tian
TL;DR
Event data is difficult for conventional vision, and ANN reconstruction can be computationally intensive. This paper proposes fully spiking EVSNN and potential-assisted PA-EVSNN models for event-based reconstruction. On public datasets, they achieve comparable performance to ANN-based models, with reported computational-efficiency gains over their ANN architectures.
Problem
Event data is not friendly to human and traditional computer vision, while ANN-based reconstruction models can be memory- and computation-intensive.
Source: §1 · p. 1Method
The paper proposes fully spiking EVSNN using LIF and membrane-potential neurons, plus PA-EVSNN with adaptive membrane-potential neurons.
Source: §1 · pp. 1–2Results
On public datasets, EVSNN and PA-EVSNN achieve comparable performance to ANN-based models; their energy consumptions are 19.36× and 7.75× more computationally efficient than their ANN architectures, respectively.
Source: §1 · p. 2Takeaways & Limitations
The results show that spiking neurons can extract temporal information and that SNNs can perform large-scale regression tasks such as event-based video reconstruction.
Source: §5 · p. 8The energy discussion frames synaptic operations as a metric for benchmarking computational energy.
Source: §4.5 · p. 7
Abstract
from arXiv · showhide
Neuromorphic vision sensor is a new bio-inspired imaging paradigm that reports asynchronous, continuously per-pixel brightness changes called `events' with high temporal resolution and high dynamic range. So far, the event-based image reconstruction methods are based on artificial neural networks (ANN) or hand-crafted spatiotemporal smoothing techniques. In this paper, we first implement the image reconstruction work via fully spiking neural network (SNN) architecture. As the bio-inspired neural networks, SNNs operating with asynchronous binary spikes distributed over time, can potentially lead to greater computational efficiency on event-driven hardware. We propose a novel Event-based Video reconstruction framework based on a fully Spiking Neural Network (EVSNN), which utilizes Leaky-Integrate-and-Fire (LIF) neuron and Membrane Potential (MP) neuron. We find that the spiking neurons have the potential to store useful temporal information (memory) to complete such time-dependent tasks. Furthermore, to better utilize the temporal information, we propose a hybrid potential-assisted framework (PA-EVSNN) using the membrane potential of spiking neuron. The proposed neuron is referred as Adaptive Membrane Potential (AMP) neuron, which adaptively updates the membrane potential according to the input spikes. The experimental results demonstrate that our models achieve comparable performance to ANN-based models on IJRR, MVSEC, and HQF datasets. The energy consumptions of EVSNN and PA-EVSNN are 19.36$\times$ and 7.75$\times$ more computationally efficient than their ANN architectures, respectively.
1. Introduction
Event-based reconstruction makes asynchronous visual data more usable, but ANN approaches can be computationally intensive. The paper introduces fully spiking and potential-assisted reconstruction models and reports ANN-comparable performance with lower computational energy.
1. Introduction
- Event data is difficult for human vision and traditional computer vision, while ANN-based reconstruction models can consume substantial memory and computation.Image reconstruction provides a visual representation of events that can also be used by conventional frame-based computer vision.Source: §1 · p. 1
- Earlier SNN work addressed tasks such as classification, optical estimation, motion segmentation, and angular-velocity regression, rather than deep SNN image reconstruction.The paper presents its work as the first attempt to develop a deep SNN for image reconstruction.Source: §1 · pp. 1–2
- EVSNN is a fully spiking event-based reconstruction framework using LIF neurons and a membrane-potential neuron.The paper identifies this as its first exploration of a fully spiking architecture for event-based image reconstruction.Source: §1 · pp. 1–2
- PA-EVSNN uses adaptive membrane-potential neurons to improve EVSNN's temporal receptive field by adjusting membrane time constants to input spikes.The adjustment is intended to adapt the network to different reconstruction scenes.Source: §1 · p. 2
- 19.36× and 7.75× more computationally efficient than their ANN architectures, EVSNN and PA-EVSNN, respectively, while achieving comparable performance to existing ANN-based models.The reported results are from experiments on public datasets.Source: §1 · p. 2
2. Related work
Related work covers approaches to SNN training and event-based reconstruction, while the paper proposes deep SNN models for reconstructing videos from event streams.
2. Related work
- Surrogate derivatives for threshold-triggered firing enabled gradient-descent training of multilayer SNNs, following earlier supervised approaches such as SpikeProp.The passage also lists Tempotron, Re-SuMe, and SPAN among earlier single-layer SNN optimization methods.Source: §2 · p. 2
- Earlier event-video reconstruction includes hand-crafted intensity estimation and deep ANN approaches, including lightweight inference, synthetic-data training, self-supervision, and CNN-transformer models.Examples include optimization, regularization, and temporal filtering, alongside learned reconstruction methods.Source: §2 · p. 2
- Prior learning-based SNN work focused mainly on classification, optical and depth estimation, motion segmentation, and angular-velocity regression; unsupervised SNN reconstruction for spike cameras was also proposed.The cited task examples include classification [62], optical estimation, motion segmentation [33], angular-velocity regression, and spike-camera reconstruction.Source: §2 · p. 2
- Figure 1 shows event streams split into windows and converted to voxel grids; the SNN uses each current temporal bin and prior membrane potentials to generate reconstructions.Each voxel grid contains N temporal bins.Source: Figure 1 · p. 2
- The paper proposes deep SNN models for reconstructing videos from event streams, unlike the ANN-based reconstruction models discussed in related work.The paper identifies this as its proposed direction for energy-efficient reconstruction.Source: §2 · p. 2
3. Method
The method converts asynchronous events into continuous voxel-grid inputs and processes them with two reconstruction SNN designs. EVSNN is fully spiking, while PA-EVSNN adds membrane-potential processing and adaptive time constants; training uses BPTT.
3.1. Input Representations
- The model converts asynchronous events into a continuous voxel grid whose bins adaptively divide the event stream and preserve normalized event timestamps.The continuous voxel grid is used to train and test the model.Source: §3.1 · p. 2
3.2. Spiking Neurons
- LIF neurons integrate inputs until threshold crossing produces a spike and reset, whereas MP neurons output membrane potential and retain temporal information without firing.The membrane time constant controls the balance between remembering the current input and forgetting the previous potential.Source: §3.2 · p. 3, Figure 2 · p. 3
3.3. The Proposed SNN Model
- EVSNN is fully spiking, while PA-EVSNN adds membrane-potential neurons to the encoder and decoder and adaptive membrane-potential neurons to extend temporal receptive fields.PA-EVSNN introduces floating-point operations through its membrane-potential neurons, so the authors describe it as a hybrid network.Source: §3.3 · pp. 3–4
- AMP neurons adjust the membrane time constant according to input spikes, using estimated global firing rate and local motion intensity to adapt to reconstruction scenes.The firing rate is estimated by average pooling, while local motion intensity is estimated from spikes using convolution and max pooling.Source: §3.3 · p. 4
3.4. Training Details of SNN
- Training unrolls the network across discrete time steps with BPTT, calculates loss every five steps, and sums time-step gradients for weight updates.The sequence length is set to 40–60 time steps; the loss combines LPIPS and temporal-consistency terms.Source: §3.3 · p. 4, §3.4 · p. 5
4. Experiments
The experiments evaluate event-based reconstruction on public datasets, compare it with ANN models, and examine temporal components, SNN architecture choices, and energy metrics.
4.1. Experimental setup
- Training uses 950 synthetic sequences and 50 validation sequences, with randomly cropped 128×128 event and image inputs.The synthetic data are generated using ESIM and the training setup follows the E2VID data source.Source: §4.1 · p. 5
4.2. Evaluation on Public Datasets
- The evaluation uses IJRR, MVSEC, and HQF, with histogram normalization and events between adjacent frames used to align reconstructions with ground truth.The models are compared with E2VID, FireNet, SPADE-E2VID, and E2VID+ using MSE, SSIM, and LPIPS.Source: §4.2 · p. 5
4.3. Temporal Component Ablation
- After zero-initialized temporal states receive continuous events through T=50, empty event inputs test how long each model retains temporal information.Across 50 IJRR sequences, E2VID, EVSNN, and PA-EVSNN improve as T increases; after T=50 their scores decline slowly, and PA-EVSNN scores better than EVSNN, though SNN temporal extraction may be weaker than ANN+LSTM.Source: §4.3 · pp. 5–6
4.2. Evaluation on Public Datasets
- 24.15× and 8.76× lower energy consumption than E2VID are reported for EVSNN and PA-EVSNN, respectively, alongside comparable performance to most ANN models.E2VID+ performs best on most datasets, while PA-EVSNN is comparable to E2VID and FireNet.Source: §4.2 · p. 5, Figure 6 · p. 6
4.4. Spiking Neural Network Architecture
- The architecture ablations favor LIF over IF and PLIF, AMP LIF among membrane-potential neurons, and CONCAT among spike connections for the selected models.ADD performs best but can produce non-spike outputs; CONCAT performs best among spike-output connections, and the chosen configurations are EVSNN-e3-res1 and PA-EVSNN-e3-res1.Source: §4.4 · pp. 6–7
4.5. Energy Consumption and Limitation Analysis
- The energy discussion identifies synaptic-operation counts as a computational-energy metric; the figure caption reports energy ratios against E2VID.The reported ratios are 24.15 times and 8.76 times lower for EVSNN and PA-EVSNN, respectively.Source: Figure 6 · p. 6, §4.5 · p. 7
5. Conclusion
The conclusion presents EVSNN and PA-EVSNN as SNN approaches to event-based video reconstruction. It reports improved computational efficiency relative to E2VID and identifies potential for low-consumption applications.
5. Conclusion
- Compared with E2VID, EVSNN and PA-EVSNN are 24.15× and 8.76× more computationally efficient, respectively.The authors state that spiking neurons can extract temporal information and that SNNs can perform large-scale regression tasks such as event-based video reconstruction.Source: §5 · p. 8