Source-linked AI summary
Spike-FlowNet: Event-based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks
Chankyu Lee, Adarsh Kumar Kosta, Alex Zihao Zhu, Kenneth Chaney, Kostas Daniilidis, Kaushik Roy
TL;DR
Event-based optical-flow estimation must handle asynchronous, discrete data that challenges conventional computer-vision methods and ANNs, while deep SNNs face spike vanishing. Spike-FlowNet combines SNN and ANN layers with a spatio-temporal event encoding and self-supervised training, and the authors report accurate flow prediction with substantial computational-efficiency benefits over a corresponding ANN architecture.
Problem
Asynchronous, discrete event-camera outputs challenge conventional methods and ANNs, while spike vanishing limits performance in deep SNNs.
Method
Spike-FlowNet integrates SNNs in initial layers and ANNs in deeper layers, using a spatio-temporal event encoding and approximate-gradient training.
Results
Spike-FlowNet accurately predicts optical flow from discrete asynchronous event streams with substantial computational-efficiency benefits compared to the corresponding ANN architecture.
Takeaways & Limitations
The reported results support hybrid SNN-ANN processing as an energy-efficient approach to event-based optical-flow estimation without sacrificing prediction performance.
Takeaways & Limitations
The encoder block accounts for 17.6% of total computations, reducing Spike-FlowNet's overall energy benefits.
Abstract
from arXiv · showhide
Event-based cameras display great potential for a variety of tasks such as high-speed motion detection and navigation in low-light environments where conventional frame-based cameras suffer critically. This is attributed to their high temporal resolution, high dynamic range, and low-power consumption. However, conventional computer vision methods as well as deep Analog Neural Networks (ANNs) are not suited to work well with the asynchronous and discrete nature of event camera outputs. Spiking Neural Networks (SNNs) serve as ideal paradigms to handle event camera outputs, but deep SNNs suffer in terms of performance due to the spike vanishing phenomenon. To overcome these issues, we present Spike-FlowNet, a deep hybrid neural network architecture integrating SNNs and ANNs for efficiently estimating optical flow from sparse event camera outputs without sacrificing the performance. The network is end-to-end trained with self-supervised learning on Multi-Vehicle Stereo Event Camera (MVSEC) dataset. Spike-FlowNet outperforms its corresponding ANN-based method in terms of the optical flow prediction capability while providing significant computational efficiency.
1 Introduction
Event cameras address limitations of frame-based vision, but their asynchronous outputs challenge conventional networks. Spike-FlowNet combines SNNs and ANNs to estimate optical flow efficiently while preserving performance.
- Frame-based cameras suffer motion blur, poor low-light capture, and saturation problems in high-dynamic-range scenes.
- Event cameras asynchronously detect pixel-level brightness changes, offering microsecond temporal resolution and lower power consumption for high-speed visual motion estimation.
- Deep SNNs exploit sparse event dynamics but can lose spikes in deeper layers, degrading performance.
- Spike-FlowNet integrates SNNs and ANNs across layers to process sparse spatio-temporal event inputs while addressing spike vanishing.
- The method uses a spatio-temporal event representation, self-supervised training, and evaluation on the MVSEC dataset against current state-of-the-art approaches.
2 Related Work
Prior optical-flow methods use conventional vision techniques or ANN adaptations for event data. SNN-based approaches instead exploit asynchronous spike timing, motivating SNNs for event-based optical-flow estimation.
- Earlier event-based optical-flow methods generally adapted conventional computer-vision techniques or deep ANNs to discrete event outputs.
- Computer-vision approaches include gradient-based Lucas-Kanade, plane fitting, frequency-based, and correlation-based methods.
- ANN approaches estimate optical flow using architectures and training modifications such as U-Net, census loss, gray-image self-supervision, brightness constraints, smoothness, and adversarial loss.
- SNNs are proposed as suitable for event outputs because precise spike timing naturally captures asynchronous spatio-temporal dynamics.
3 Method
Spike-FlowNet processes fine-grained event representations with an SNN encoder and ANN residual-decoder blocks, trained end-to-end using self-supervised photometric and smoothness losses. Its hybrid design addresses vanishing spikes in deep SNNs while retaining event-driven processing for sparse inputs.
- 3.1 Spiking Neuron Model: The Integrate and Fire neuron integrates weighted input spikes into membrane potential, emits a spike at threshold crossing, and then resets.This recurrent thresholding mechanism supports event-based computation across SNN layers.
- 3.2 Spiking input event representation: Each event is represented as an Address Event Representation 4-tuple containing pixel coordinates, timestamp, and brightness-change polarity.Event cameras generate these discrete events asynchronously when log-scale intensity changes exceed a threshold.
- 3.2 Spiking input event representation: Spike-FlowNet discretizes each time window into former and latter groups of N accumulated event frames, each with separate ON and OFF channels.Frames are passed sequentially, preserving spatial and temporal information rather than feeding all event information forward at once.
- 3.3 Self-Supervised Loss: Self-supervised training combines photometric reconstruction and smoothness losses using grayscale images captured at the beginning and end of each event window.Predicted flow warps the second grayscale image toward the first, while smoothness regularizes neighboring flow estimates.
- 3.4 Spike-FlowNet Architecture: The hybrid architecture places the encoder in the SNN block and the residual and decoder layers in the ANN block to limit vanishing spike effects in deep networks.Its U-Net-like topology has four encoder layers, two residual blocks, and four decoder layers, with encoder outputs accumulated over time.
- 3.5 Backpropagation Training in Spike-FlowNet: Training uses standard backpropagation through the ANN block and approximate surrogate gradients through the SNN layers.The SNN is unrolled over discrete time steps using BackPropagation Through Time, enabling end-to-end self-supervised training.
4 Experimental Results
Spike-FlowNet is evaluated on MVSEC across indoor and outdoor sequences using AEE, optical-flow visualizations, and comparisons with prior methods. Its sparse event processing also yields substantial computational-energy advantages, although the encoder accounts for only 17.6% of total computations.
- Evaluation setup: MVSEC evaluation uses outdoor day2 for training and indoor flying 1,2,3 plus outdoor day1 for testing.This protocol follows prior work for fair comparison; testing uses center-cropped 256 × 256 left-camera images.
- Evaluation metric: AEE measures the mean distance between predicted and ground-truth optical flow at active pixels, evaluated for dt = 1 and dt = 4.Only pixels containing both events and ground-truth flow contribute to the metric.
- Optical-flow results: Spike-FlowNet accurately predicts optical flow indoors and outdoors, with sparse predictions in low-texture regions where few events occur.Figure 5 visualizes grayscale images, spike events, ground-truth flow, predicted flow, and masked flow for outdoor day1 and indoor day1.
- Optical-flow results: Spike-FlowNet outperforms EV-FlowNet in AEE for both dt = 1 and dt = 4, while matching or slightly exceeding Zhu et al. at dt = 4 but underperforming at dt = 1.The comparison with EV-FlowNet is described as fair because both use similar architectures and self-supervised learning.
- Computational efficiency: 0.48% and 1.01% are the mean spike activities for dt = 1 and dt = 4, respectively, enabling sparse event-based computation in SNN layers.ANNs instead perform dense operations, including on zero and non-zero inputs.
- Computational efficiency: 214.2× and 25.51× are the encoder-block computational-energy efficiencies over ANN-based counterparts for dt = 1 and dt = 4, respectively.The larger dt = 1 benefit is attributed to its four-times-smaller number of time-steps; the encoder represents 17.6% of total computations.
5 Conclusion
Spike-FlowNet combines SNNs and ANNs for energy-efficient optical-flow estimation from event-camera data. Its encoding and self-supervised training support accurate prediction with substantial computational-efficiency benefits over a corresponding ANN architecture.
- 5 Conclusion: Spike-FlowNet integrates SNNs and ANNs across layers to address spike vanishing in deep SNNs.Its input encoding preserves spatial and temporal information from event-camera outputs.
- 5 Conclusion: The network uses self-supervised learning, avoiding the need for expensive labeling during training.The reported system estimates optical flow from discrete, asynchronous event streams.
- 5 Conclusion: Spike-FlowNet accurately predicts optical flow while providing substantial computational-efficiency benefits compared with the corresponding ANN architecture.
6 Ablation Study
The ablation study evaluates hybrid topology, event-input grouping, and smoothness-loss weighting using Average Endpoint Error (AEE). Results favor the Spike-FlowNet design, N = 2 input groups, and different λ values depending on dt.
- 6.1 Hybrid Network: AEE degrades as more residual blocks are converted to SNN layers, with Spike-FlowNet achieving the best results for dt = 1 and dt = 4.The authors attribute this degradation to spike vanishing aggravating with network depth.
- 6.2 Input representation: N = 2 input groups provides optimal AEE results for both dt = 1 and dt = 4.Increasing N worsens dt = 1 performance, while dt = 4 converges to a reasonably accurate flow estimate.
- Ablation overview: Table 3 reports Average Endpoint Error (AEE) for ablations across different network topologies, input representations, and loss-weight choices.These comparisons test the optimal design choices of the hybrid network, input representation, and smoothness-loss weight λ.
- 6.3 Smoothness-loss weight: λ = 10 and λ = 100 produce more accurate dt = 1 flow estimates, whereas λ = 1 works better for dt = 4.The authors relate the higher smoothness requirement for dt = 1 to its sparser, more discontinuous flow outputs.