Source-linked AI summary
AEGNN: Asynchronous Event-based Graph Neural Networks
Simon Schaefer, Daniel Gehrig, Davide Scaramuzza
TL;DR
Event-camera algorithms often lose sparsity and temporal resolution by converting events into dense representations, increasing computational demands and latency. AEGNNs instead process events as evolving spatio-temporal graphs with localized asynchronous updates, while retaining synchronous training; across recognition and detection, they achieve competitive performance with substantially reduced computation.
Problem
Dense event representations discard event sparsity and asynchrony, causing wasteful computation, while conventional updates recompute unchanged activations.
Method
AEGNNs process events as temporally evolving graphs and asynchronously propagate local activation updates, while allowing synchronous batch training and later asynchronous deployment.
Results
Across object recognition and detection, AEGNNs show similar performance to state-of-the-art methods while requiring up to 11 times less compute.
Takeaways & Limitations
The method provides sparse, asynchronous event processing with reduced computational effort while preserving the events' temporal resolution.
Takeaways & Limitations
CNN implementations are highly optimized, so the reported computation reduction translates to latency gains only on suitable hardware such as FPGAs or IPUs.
Abstract
from arXiv · showhide
The best performing learning algorithms devised for event cameras work by first converting events into dense representations that are then processed using standard CNNs. However, these steps discard both the sparsity and high temporal resolution of events, leading to high computational burden and latency. For this reason, recent works have adopted Graph Neural Networks (GNNs), which process events as ``static" spatio-temporal graphs, which are inherently "sparse". We take this trend one step further by introducing Asynchronous, Event-based Graph Neural Networks (AEGNNs), a novel event-processing paradigm that generalizes standard GNNs to process events as ``evolving" spatio-temporal graphs. AEGNNs follow efficient update rules that restrict recomputation of network activations only to the nodes affected by each new event, thereby significantly reducing both computation and latency for event-by-event processing. AEGNNs are easily trained on synchronous inputs and can be converted to efficient, "asynchronous" networks at test time. We thoroughly validate our method on object classification and detection tasks, where we show an up to a 11-fold reduction in computational complexity (FLOPs), with similar or even better performance than state-of-the-art asynchronous methods. This reduction in computation directly translates to an 8-fold reduction in computational latency when compared to standard GNNs, which opens the door to low-latency event-based processing.
Multimedia Material
The project page provides videos, code, and additional materials for AEGNN.
- Videos, code, and more are available on the AEGNN project page.
1. Introduction
Event cameras provide sparse, asynchronous, high-temporal-resolution data, but conventional dense processing wastes computation. AEGNNs address this by processing events as evolving graphs with asynchronous updates and achieve competitive performance with substantially lower compute.
- Filtering-based methods provide low latency but rely on handcrafted equations that do not scale well to complex tasks such as detection or classification.
- Dense image-like event representations discard sparsity and asynchrony, causing wasteful computation and higher latency and power consumption.
- AEGNNs model events as temporally evolving spatio-temporal graphs and update only activations affected by each new event.
- AEGNNs can be trained on synchronous batches and deployed asynchronously at test time using general update rules applicable to most GNN architectures.
- Up to 11 times less compute is required while maintaining similar performance to state-of-the-art methods; object detection shows 32% computation reduction with up to a 3.4% mAP increase.
2. Related Work
Prior sparse and geometric event-vision methods reduce dense computation but retain limitations in temporal modeling, hierarchical learning, or activation recomputation. AEGNNs combine graph-based processing with recursive asynchronous updates to address these gaps.
- CNN-based event-vision models achieve state-of-the-art performance but ignore event sparsity and asynchrony, causing redundant computation.
- Graph-based methods reduce complexity by restricting message passing to sparse event connections, but batch processing still recomputes all activations for each new event.
- Existing approaches either lack hierarchical learning, discard temporal information through image-like inputs, or use shallow embeddings without end-task learning.
- AEGNNs combine graph-based methods with recursive update rules, enabling multilayer learning while better exploiting spatio-temporal event sparsity.
3. Prerequisites
The method represents events as nodes in a spatio-temporal graph and applies graph convolutions, pooling, and a prediction head. Graph convolutions aggregate neighborhood messages, while pooling coarsens the graph by clustering nodes.
- Events become graph nodes, neighboring events form directed edges, and a GNN maps the resulting graph G to a prediction y.
- Graph Convolutions: Graph convolutions compute pairwise neighbor messages, aggregate them, and transform the aggregate into a new node value.
- Graph Pooling: Graph pooling transforms graph G into a coarser graph Gc by aggregating nodes into clusters centered at selected nodes.
- Graph Pooling: Cluster-based pooling can use symmetric aggregation functions such as summation, maximum, or minimum, and requires reconnecting edges after clustering.
- Processing Pipeline: The illustrated detection pipeline subsamples events, constructs a sparse spatio-temporal graph, processes it with a GNN, and predicts bounding boxes.
4. Approach
AEGNN represents event streams as evolving spatio-temporal graphs and updates only the graph regions affected by incoming events. The approach combines sparse graph construction, localized asynchronous propagation, and sparse updates across selected network operations.
- Graph Construction: Events are embedded in spatio-temporal space and uniformly subsampled by K = 10 before graph construction.Each remaining event becomes a graph node; temporal coordinates are normalized to align with spatial coordinates.
- Graph Construction: Edges connect event nodes within spatio-temporal distance R, with neighborhood size limited to Dmax and features assigned from polarity and relative position.The initial node feature is event polarity, while edge features encode normalized relative positions.
- Asynchronous Processing: New events add one node and sparse local connections, so the first graph layer recomputes only the new node’s 1-hop subgraph.The resulting activation changes are propagated through subsequent layers rather than recomputing the entire graph.
- Asynchronous Processing: At layer N, updates cover the N-hop neighborhood HN(i′), making computation proportional to the affected subgraph rather than the full graph.The neighborhood size depends on graph connectivity and determines the computation required at each layer.
- Sparse Network Operations: Sparse graph convolutions and pooling use initialization and processing stages, while non-graph layers provide only small efficiency gains because many nodes still require updates.Spline convolutions are selected as a balance between computational complexity and predictive accuracy, while retaining relative positional information.
- Network Details: The model contains 7 graph convolution blocks and 2 pooling layers, and achieves state-of-the-art object-recognition performance with the lowest overall computational complexity.On N-Cars, it obtains the best accuracy with 20 times lower computational complexity than the second-best asynchronous method.
5. Experiments
Experiments evaluate AEGNN for object recognition and detection, emphasizing predictive performance, computational efficiency, scalability, and timing. Across these tasks, AEGNN achieves competitive detection and recognition results while reducing computation and update latency.
- Experimental setup: AEGNN is evaluated on object recognition and object detection using the PyG and Torch frameworks, with Adam-based training.The experiments apply AEGNN to both tasks and use Lightning for training.
- 5.1. Object Recognition: Recognition experiments compare AEGNN with asynchronous and synchronous methods using test accuracy and FLOPs per additional event.The comparison processes windows of 25,000 events and measures the operations required to update each event.
- 5.1. Object Recognition: 300-fold computation reduction is reported against the asynchronous method [36], while 5,000 events suffice for state-of-the-art recognition accuracy.The savings are attributed to the flat architecture and sparse graph representation.
- 5.2. Object Detection: For detection, AEGNN outperforms NVS-S by 7.7% using 21 times less computation, while using 0.39 MFLOPs/ev versus 4712 MFLOPs/ev for RED.Against AsyNet, performance varies by dataset: AsyNet is higher on N-Caltech101, whereas AEGNN is 3.4 mAP higher on Gen1 with 520–540 times fewer MFLOPs per event.
- Timing experiments: For 25,000 events, the sparse update takes 129ms versus 1014ms for a dense GNN, an 8-fold improvement.For one event in a 4,000-node N-Caltech101 graph, the sparse update takes 92ms versus 167ms for the dense update.
- Timing experiments: The measured implementation is only 1.5 times faster than a CNN, whose PyTorch implementation is highly optimized.The authors expect suitable FPGA or IPU hardware to better translate computation reduction into latency and power savings.
6. Conclusion
AEGNNs model events as evolving spatio-temporal graphs and update only affected nodes, reducing computation for recognition and detection while retaining or improving performance. The reported reductions include up to 11-fold fewer FLOPs, 32% lower detection computation, and eightfold lower latency than dense GNNs.
- 6. Conclusion: AEGNNs restrict recomputation to affected nodes and propagate updates to lower layers for event-by-event processing.The networks model events as evolving spatio-temporal graphs and support batch training followed by asynchronous deployment.
- 6. Conclusion: 11-fold lower computational complexity (FLOPs) was achieved for object recognition.
- 6. Conclusion: 32% lower computational complexity and 3.4% higher mAP than asynchronous methods were achieved for object detection.
- 6. Conclusion: 8-fold lower processing latency was achieved compared to dense GNNs.
- 6. Conclusion: Specialized FPGA or IPU hardware is expected to further reduce latency and power consumption.The conclusion states that these additional reductions are expected if the method is implemented on specialized hardware.
8. Appendix
The appendix documents the network architectures, Spline Convolution formulation, FLOPS accounting, and dataset licenses used in the work.
- Network architectures: Both recognition and detection networks use convolutional blocks containing SplineConv, ELU activation, and batch normalization.Max graph pooling follows the fifth and seventh convolutional layers.
- Spline Convolutions: Spline Convolutions incorporate neighbors’ spatial arrangement through pseudo-coordinates, producing richer features than standard GNN aggregation.Pseudo-coordinates are normalized distance vectors between neighboring nodes; the spline kernel is expanded over a three-dimensional regular grid with eight elements and learnable weights.
- FLOPS computation: The FLOPS derivation computes pseudo-coordinates, B-Spline bases, kernel terms, neighbor aggregation, and output normalization.It defines Ni = |N(i)| and Np = |P|, and counts the operations required for a single-node Spline Convolution.
- FLOPS computation: The resulting Spline Convolution FLOPS expression combines feature-channel, neighborhood, kernel-grid, and spatial-dimension costs.
- Dataset licenses: The appendix lists licenses and sources for the Prophesee Gen1, N-Caltech101, and N-Cars datasets.