Source-linked AI summary
Event-based Asynchronous Sparse Convolutional Networks
Nico Messikommer, Daniel Gehrig, Antonio Loquercio, Davide Scaramuzza
TL;DR
Event-based vision needs methods that extract information from sparse, asynchronous event streams without the redundant computation and latency of synchronous processing. The paper converts models trained on synchronous event representations into asynchronous networks with identical outputs, and reports reduced computation alongside competitive or improved accuracy across recognition and detection tasks.
Problem
Efficiently extracting information from event streams remains challenging because synchronous processing discards event sparsity and performs redundant computation at higher latency.
Method
The framework converts synchronous neural networks operating on image-like event representations into asynchronous networks that update computation locally around incoming events.
Results
The method reduces computational complexity up to 20 times, outperforms synchronous models by up to 15% at equal FLOP budgets, and asynchronous approaches by up to 24.5% in accuracy.
Takeaways & Limitations
The framework combines low-latency asynchronous processing with high-accuracy neural networks while explicitly exploiting spatio-temporal event sparsity.
Takeaways & Limitations
For typical channel sizes cl ≥16, the framework incurs less than 3% approximation error.
Abstract
from arXiv · showhide
Event cameras are bio-inspired sensors that respond to per-pixel brightness changes in the form of asynchronous and sparse "events". Recently, pattern recognition algorithms, such as learning-based methods, have made significant progress with event cameras by converting events into synchronous dense, image-like representations and applying traditional machine learning methods developed for standard cameras. However, these approaches discard the spatial and temporal sparsity inherent in event data at the cost of higher computational complexity and latency. In this work, we present a general framework for converting models trained on synchronous image-like event representations into asynchronous models with identical output, thus directly leveraging the intrinsic asynchronous and sparse nature of the event data. We show both theoretically and experimentally that this drastically reduces the computational complexity and latency of high-capacity, synchronous neural networks without sacrificing accuracy. In addition, our framework has several desirable characteristics: (i) it exploits spatio-temporal sparsity of events explicitly, (ii) it is agnostic to the event representation, network architecture, and task, and (iii) it does not require any train-time change, since it is compatible with the standard neural networks' training process. We thoroughly validate the proposed framework on two computer vision tasks: object detection and object recognition. In these tasks, we reduce the computational complexity up to 20 times with respect to high-latency neural networks. At the same time, we outperform state-of-the-art asynchronous approaches up to 24% in prediction accuracy.
Multimedia Material
The project provides public implementation code and a video with additional qualitative results.
- The project code is available online.The passage provides the project repository URL.
- Additional qualitative results are available in a video.The passage provides a video URL for viewing them.
- The supplied materials include both implementation resources and qualitative-result media.
1 Introduction
Event cameras produce sparse, asynchronous brightness-change signals, creating a challenge for low-latency information extraction. Existing approaches either require substantial task-specific engineering or perform redundant computation, while this work proposes a general sparse asynchronous framework for high-performance networks.
- Event cameras measure per-pixel brightness changes asynchronously rather than capturing dense images at fixed rates.Their signals are sparse in space and time and usually encode moving image edges.
- Efficiently extracting information from event streams requires exploiting their spatio-temporal sparsity while maintaining minimal latency.
- Filtering-based methods provide high temporal resolution and low latency but require handcrafted features and update functions.This engineering burden makes generalization to tasks such as object recognition and detection difficult.
- SNN denotes continuous-time neural networks in the neuromorphic-literature usage adopted here.The passage distinguishes this usage from low-precision networks sometimes also called SNNs.
- Processing redundant computation increases inference time and sacrifices the event signal’s inherent low-latency property.
- The proposed framework converts synchronous networks into asynchronous models that exploit sparse event inputs without changing training.It is described as agnostic to event representation, architecture, and task, and is shown to produce identical results with less computation.
2 Related Work
Event-based vision has increasingly adopted deep, data-driven models because they achieve strong results, but their conventional processing can discard event sparsity and increase latency. Sparse convolution offers one response to this inefficiency.
- Deep neural networks have increasingly replaced handcrafted filtering methods in event-based learning.
- Data-driven approaches achieve state-of-the-art results across low- and high-level vision tasks.
- Their high performance can come at the cost of discarding event sparsity and asynchrony, causing redundant computation, higher latency, and greater bandwidth.
- Spatially sparse convolutional techniques have been proposed to process sparse event-derived data more efficiently.
3 Method
The method converts synchronous image-like event-processing networks into asynchronous models by updating only event-induced changes while preserving the network’s outputs. It combines sparse representations, submanifold sparse convolutions, and incremental rulebook propagation to reduce computation as event updates move through the network.
- 3.1 Event Data: Event representations are formed from asynchronous event sequences for processing by standard CNNs, but synchronous reprocessing discards temporal sparsity and creates redundant computation.
- 3.1 Event Data: Sparse recursive representations update only a few positions when each new event arrives; histograms, event queues, and time images can update a single pixel.
- 3.2 Exploiting the Sparsity of the Event Signal: Submanifold sparse convolutions compute only at active, non-zero sites, preserving the same active-site pattern and sparsity across layers at one spatial resolution.
- Asynchronous processing: The asynchronous framework propagates event-induced changes through layers by incrementally building rulebooks and receptive fields, while updating newly active or inactive sites locally.
- Asynchronous processing: Processing events sequentially produces the same result as processing them together, allowing a trained synchronous network to be deployed asynchronously without changing optimization or learning.
- Computational Complexity: A fractal-dimension slope γ < 2 indicates slower-than-quadratic growth of update sites, making computation increasingly efficient as the event data’s fractal dimension decreases.
4 Experiments
Experiments on object recognition and detection show that the asynchronous sparse framework preserves or improves accuracy while substantially reducing computation. It supports multiple event representations and achieves strong comparisons against dense and asynchronous methods.
- Experimental Setup: The framework was validated on object recognition and object detection using event histogram and event queue representations.Both representations can be updated sparsely and asynchronously for each incoming event.
- Object Recognition: 19.5 times less computation per event was required for similar classification accuracy than dense synchronous processing.Both methods used the same VGG13 architecture and input representation generated with 25,000 events.
- Object Recognition: 14.8% better accuracy was achieved under a fixed computation budget than a standard CNN.The asynchronous method retains state and performs sparse updates, whereas standard networks recompute all activations for each event.
- Object Recognition: 4.3% higher accuracy on N-Caltech101 and 1.7% on N-Cars were achieved than YOLE at 6% of its average computational cost.The comparison covers state-of-the-art asynchronous and lightweight per-event-update methods.
- Object Detection: 10.6 times fewer FLOPs per event were used than dense processing for object detection, averaged across datasets and representations.On N-Caltech101, histogram processing improved accuracy by 2.0%, while Gen1 Automotive showed a 1.8% performance drop.
- Object Detection: 24.5% higher accuracy was achieved than YOLE at 5% of its computational cost for object detection.The method achieved comparable or superior performance to standard networks at a fraction of their computational cost.
5 Discussion
The discussion argues that synchronous batch processing has overlooked event data’s intrinsic sparsity and asynchrony. Reintroducing these properties into high-performance CNNs reduces computation while improving accuracy at matched computational budgets.
- Discussion: Up to 20 times lower computational complexity was achieved by bringing spatio-temporal event sparsity into high-performance CNNs.The paper contrasts this with event-based methods that process events synchronously in batches.
- Discussion: Up to 15% better accuracy than state-of-the-art synchronous models was achieved at the same FLOP budget.The comparison is reported at matched computational cost.
- Discussion: Existing asynchronous approaches were outperformed by up to 24.5% in accuracy.The discussion presents this as evidence for incorporating event data’s intrinsic properties into low-latency, high-accuracy algorithms.
7 Supplementary Material
The supplementary material describes recursive rulebook construction and asynchronous propagation of event-induced changes through network layers, then proves equivalence with synchronous outputs.
- The input layer initializes with an empty rulebook and a receptive field containing the initial active site.
- Each layer reuses the previous rulebook and receptive field by separating newly reached frontier sites from already visited sites.The receptive field is decomposed into a frontier set and a visited state set.
- Newly active and inactive sites update the rulebook, while their outputs are computed directly or set to zero before propagating remaining contributions.
- New events produce non-zero increments only at affected input sites, including sites that remain active, become inactive, or become active.
- The update procedure repeats layer by layer using the changed sites from the preceding layer, completing the equivalence proof.
7.3 Representations and FLOP computation
The framework does not require a specific input representation and evaluates event embeddings that remain sparse in time and space.
- The asynchronous framework is agnostic to the input representation.
- The evaluated event embeddings are sparse in both time and space.
- Two event embeddings are evaluated for both tasks.
Representations
The experiments use event histograms and event queues as sparse temporal-spatial representations; the histogram encodes polarity over a fixed-size event window.
- The two tested representations are the event histogram and the event queue.
- The event histogram forms a two-channel image by separating positive- and negative-polarity events.
- A constant-event sliding window creates an update site whenever an event enters or leaves the histogram.
FLOP computation
FLOP computation compares standard dense convolution with the asynchronous sparse formulation using layer-specific output dimensions, active sites, and rule counts.
- Table 5 reports FLOPs for different network operations in standard networks and the asynchronous sparse method.
- A standard convolution requires HoutWoutcout(2k^2cin −1) FLOPs excluding bias.This counts k^2cin multiplications and k^2cin −1 additions per output pixel and channel.
- The asynchronous sparse formulation computes operations by following Eqs. (M-8) and (M-9), using the number of rules at each layer.
7.4 Sensitivity on the Number of Events
The study evaluates sparse and dense VGG13 across varying event counts on N-Caltech101. Sparse-network accuracy peaks at 25’000 events, where computation is lower than for the dense network.
- 25’000 events maximizes test accuracy for the sparse network, while dense-network accuracy reaches a plateau.
- 46% lower computation is achieved by the sparse network than the dense network at 25’000 events.
- Table 6 reports computational complexity in MFLOPS and test accuracy for sparse and dense VGG13 across varying event counts.
7.5 Qualitative Results on Object Detection
Qualitative object-detection results show successful detections alongside failures involving visually similar classes and cars generating few events. The figure distinguishes success cases in its first two columns from a failure case in the last column.
- Similar classes, including pigeons and roosters, are a principal source of detection failures.
- Cars with high relative motion are detected well, but cars moving slowly can be missed because they trigger few events.
- The first two columns show success cases, while the last column shows a failure case, with predictions in magenta and labels in cyan.