Source-linked AI summary
Asynchronous Convolutional Networks for Object Detection in Neuromorphic Cameras
Marco Cannici, Marco Ciccone, Andrea Romanoni, Matteo Matteucci
TL;DR
Object detection on event-based cameras remains supported by few datasets and algorithms. The paper proposes YOLE, which processes events through surfaces, and fcYOLE, an asynchronous fully convolutional approach that exploits event sparsity. YOLE achieves good detection results on most datasets, while event-based inference is faster on one timing benchmark and slower on another.
Problem
Object detection in event streams has few available datasets and proposed algorithms.
Source: §1 · p. 1Method
Results
YOLE achieves good detection results on most datasets, while event-based inference is 2x faster on Shifted N-MNIST and slightly slower on Blackboard MNIST.
Source: §4.3 · pp. 6–7Takeaways & Limitations
The asynchronous framework can be applied to fully convolutional architectures and can adapt computation to incoming events while maintaining internal state.
Source: §5 · p. 8Performance on N-Caltech101 is poor in aggregate as class sample counts are imbalanced and performance worsens with fewer training samples and greater within-class variability.
Source: §4.3 · p. 7
Abstract
from arXiv · showhide
Event-based cameras, also known as neuromorphic cameras, are bioinspired sensors able to perceive changes in the scene at high frequency with low power consumption. Becoming available only very recently, a limited amount of work addresses object detection on these devices. In this paper we propose two neural networks architectures for object detection: YOLE, which integrates the events into surfaces and uses a frame-based model to process them, and fcYOLE, an asynchronous event-based fully convolutional network which uses a novel and general formalization of the convolutional and max pooling layers to exploit the sparsity of camera events. We evaluate the algorithm with different extensions of publicly available datasets and on a novel synthetic dataset.
1. Introduction
Neuromorphic cameras provide sparse, asynchronous event streams, while object detection on these devices has had few datasets and algorithms. This paper introduces a hybrid approach that retains conventional neural-network strengths while computing features sparsely and asynchronously.
1. Introduction
- Conventional computer-vision networks commonly take images as input, motivating event-specific ways to process camera data.CNNs are widely used across image classification, detection, and scene labeling.Source: §1 · p. 1
- Neuromorphic cameras emit events only when brightness changes, yielding sparse scene updates with microsecond resolution and low power and bandwidth requirements.Each event records position, timestamp, and brightness-change polarity.Source: §1 · p. 1
- Spiking neural networks handle events directly, but training deep models on complex tasks is usually difficult; other approaches integrate events into frames or use time surfaces, memory cells, or LSTMs.The paper contrasts its approach with these routes to event processing.Source: §1 · p. 1
- Object detection in event streams has few available datasets and proposed algorithms.The paper frames this as a limited evidence base for an increasingly popular sensor type.Source: §1 · p. 1
- The proposed hybrid framework computes features sparsely while retaining conventional neural-network advantages, with computation triggered by events and limited to locations needing recomputation.It reformulates convolution and max pooling with internal state to sparsely recompute feature maps.Source: §1 · pp. 1–2
2. Background
The background introduces a leaky surface as a continuous event representation and YOLE as a frame-based detection baseline. Because YOLE does not exploit event sparsity, the section motivates an asynchronous alternative.
2. Background
- A leaky surface accumulates sparse camera events while retaining memory of past events.Each event increments its pixel, while the surface decays according to elapsed time.Source: §2 · p. 2
- The surface uses Δincr for event increments and λ for decay; the paper fixes Δincr = 1 and varies λ by dataset.The two parameters have related effects, and pixel values are clamped to prevent negative values.Source: §2 · p. 2
- Unlike fixed-interval binary frame integration, continuous incremental integration preserves time resolution and is described as better able to handle noise.The comparison is with a procedure that assigns equal importance to events within each interval.Source: §2 · p. 2
- YOLE combines YOLO training with leaky surfaces as a reference model, but it does not exploit event sparsity.The paper uses a simpler model structure and presents the asynchronous framework as the response to this limitation.Source: §2 · p. 2
- The paper chooses a simpler event representation than time surfaces or event histograms to derive a linear dependence between consecutive surfaces for its event-based framework.The cited alternatives are described as better representing complex scene dynamics.Source: §2 · p. 2
3. Event-based Fully Convolutional Networks
The section replaces full-frame feature recomputation with an asynchronous forward pass that updates features where events affect them. Stateful event-based layers and pooling support sparse inference in fcYOLE.
3. Event-based Fully Convolutional Networks
- Feature maps retain state and update when events arrive; layer-wise leaking can also update outputs at regular rates when no new events arrive.The network can instead produce outputs only when new events arrive, adapting to their timing.Source: §3 · p. 3
3.1. Leaky Surface Layer
- The leaky surface layer forwards incoming events, the leak amount, and pixels reset by max operations so subsequent layers can update their states.Source: §3.1 · p. 3
3.2. Event-based Convolutional Layer (e-conv)
- The e-conv layer recomputes affected features and stores an update map to track how input changes propagate through earlier activation functions.The update map is revised locally when an activation changes its coefficient, such as a ReLU crossing between inactive and active regions.Source: §3.2 · pp. 3–4
3.3. Event-based Max Pooling Layer (e-max-pool)
- The e-max-pool layer stores each receptive field’s maximum position and recomputes maxima when events affect a field or its previous maximum may no longer hold.Differing leak rates can shift a maximum even without an event in that region; a maximum with the minimum update rate remains maximal.Source: §3.3 · pp. 4–5
3.4. Event FCN for Object Detection (fcYOLE)
- fcYOLE converts YOLE to a fully convolutional detector, replacing fully connected layers with 1 × 1 e-conv layers; shared subnetworks can process larger surfaces without redesign or retraining.The architecture divides the 128×128 field of view into 4 × 4 regions, each predicting 2 bounding boxes.Source: Figure 1 · p. 3, §3.4 · p. 5
4. Experiments
YOLE achieves good detection results on most datasets, while fcYOLE trades a slight performance decrease for an all-event-based network with fewer parameters. Results also show sensitivity to noisy inputs, class sample counts, and event sparsity.
4.1. Datasets
- The evaluation covers modified N-MNIST and MNIST-DVS datasets, OD-Poker-DVS, synthetic Blackboard MNIST, and N-Caltech101.The N-MNIST and MNIST-DVS variants support object detection; N-Caltech101 is the only publicly available event-based dataset in this account with bounding-box annotations.Source: §4.1 · pp. 5–6
4.2. Experiments Setup
- MNIST networks use a LeNet-inspired six-layer feature extractor, while N-Caltech101 uses a simplified VGG16-inspired architecture; training uses pretrained first four convolutional layers and Adam.Dataset-specific batch sizes and validation-based early stopping were used.Source: §4.2 · p. 6
4.3. Results and Discussion
- YOLE achieves good detection results in mAP and accuracy on most datasets; fcYOLE has a slight performance decrease compared with YOLE.Table 3 compares the two networks; fcYOLE removes fully connected layers to use only event-based layers and fewer parameters.Source: §4.3 · pp. 6–7, Table 3 · p. 7
- On Shifted N-MNIST, YOLE detects multiple objects, but added clutter and random events lower accuracy and mAP.Table 4 covers single-digit, two-digit, cluttered (v2fr), and cluttered-plus-noise (v2fr+ns) variants.Source: §4.3 · p. 6, Table 4 · p. 7
- In experiments on the Blackboard MNIST and Poker-DVS datasets, using λcoord = 25.0 and λnoobj = 0.25 instead of the hyperparameters suggested by the original work [41] worked better, increasing both accuracy and mean average precision scores (v2*).Source: §4.3 · p. 7
- N-Caltech101 results are weak overall: performance worsens with fewer training samples and greater within-class variability.YOLE performs better for classes with many samples or similar samples; the text attributes the poor aggregate scores to task difficulty and class imbalance.Source: §4.3 · p. 7, Table 1 · p. 7, Table 2 · p. 7
- Removing fully connected layers gives fcYOLE only event-based layers and significantly fewer parameters, but slightly lowers performance relative to YOLE.Keeping those layers would match YOLE performance, but event-based layers are formalized only up to the first fully connected layer.Source: §4.3 · p. 7
- 2× speedup (22.6 ms per batch) was measured on Shifted N-MNIST, while Blackboard MNIST took 43.2 ms per batch versus 34.6 ms for conventional layers.Asynchronous CNNs were faster only up to 80% event sparsity; the implementation was not optimized for noisy scenes.Source: §4.3 · p. 7
5. Conclusions
The paper concludes with two YOLO-based approaches for event-camera object detection: YOLE integrates events into a leaky surface, while fcYOLE processes them asynchronously. Timing analysis is described as promising, with runtime adaptation and hardware implementation identified as future directions.
5. Conclusions
- fcYOLE exploits event sparsity through asynchronous updates, while YOLE integrates events into a leaky surface; the framework is presented as applicable to fully convolutional architectures.The detector updates in response to incoming events and maintains its internal state when none arrive.Source: §5 · p. 8
- The authors report promising timing results for the proposed formalization.They note that an ad-hoc hardware implementation could better exploit the method's advantages and allow a fair timing comparison with SNNs.Source: §5 · p. 8
- Future work includes detecting at runtime whether event-based layers or full feature-map recomputation is more beneficial.The authors also identify ad-hoc hardware implementation as a way to better exploit the method and enable fair timing comparisons with SNNs.Source: §5 · p. 8
Supplementary material
The supplementary document describes the event-based datasets used in the paper. It lists the authors and their Politecnico di Milano affiliation.
Supplementary material
- The listed authors are Marco Cannici, Marco Ciccone, Andrea Romanoni, and Matteo Matteucci.Their listed affiliation is Politecnico di Milano, Italy.Source: Supplementary material · p. 11
- The document provides email addresses for the four listed authors.The addresses use the polimi.it domain.Source: Supplementary material · p. 11
- The supplementary document describes the novel event-based datasets adopted in the paper.It identifies the associated paper as “Asynchronous Convolutional Network for Object Detection in Neuromorphic Cameras”.Source: Supplementary material · p. 11
1. Event-based object detection datasets
The paper extends N-MNIST, MNIST-DVS, and Poker-DVS for event-based object detection and introduces Blackboard MNIST, a synthetic dataset. The collections vary in object count, scale, scene composition, and camera motion.
1. Event-based object detection datasets
- Four event-based detection datasets are introduced: extensions of N-MNIST, MNIST-DVS, and Poker-DVS, plus the new Blackboard MNIST dataset.The work is motivated by the lack of object-detection datasets for event cameras.Source: §1 · p. 11
1.1. Shifted N-MNIST
- Shifted N-MNIST v1 and v2 contain one or two non-overlapping 34 × 34 digits randomly positioned on larger surfaces ranging from 68 × 68 to 124 × 124.Bounding boxes are estimated by filtering event-integrated frames, clustering remaining pixels, and removing small boxes associated with low event activity during camera stillness; the described parameters are ρ = 3, R = 2, and minarea = 10.Source: §1.1 · p. 11
1.2. Shifted MNIST-DVS
- Shifted MNIST-DVS mixes digits from three scales in a fixed 128 × 128 field of view and contains 30,000 samples across four specified object configurations.The configurations use three scale4 digits, two scale8 digits, two scale4 digits with one scale8 digit, or one scale16 digit.Source: §1.2 · pp. 11, 13
1.3. OD-Poker-DVS
- OD-Poker-DVS extends Poker-DVS from recognition to detection and contains 292 short examples, split into 218 training and 74 testing samples.Examples are about 1.5 ms long and are divided to keep the number of ground-truth boxes approximately even; the source dataset is described as a limited collection of real-world recordings.Source: §1.3 · p. 13
1.4. Blackboard MNIST
- Blackboard MNIST places MNIST digits at random positions and scales on a blackboard, addressing the predefined-size constraint of the earlier MNIST-based datasets.The simulator generates events from rendered frames using log-intensity threshold crossings and piecewise-linear interpolation, and outputs bounding-box annotations.Source: §1.4 · p. 14
- Blackboard MNIST varies camera paths and digit placement; its EASY collection uses one digit size and a straight camera trajectory, while MEDIUM varies paths, positions, and object configurations.Triangular and curved trajectories were introduced because sudden camera movements produce event bursts the detection system was intended to handle.Source: §1.4 · p. 14
- The Blackboard MNIST dataset combines three collections and contains 3,000 samples: 2,500 training, 250 testing, and 250 validation.The described collections are EASY, MEDIUM, and HARD.Source: §1.4 · pp. 14–15
2. Results
The results section points to a comparison of YOLE and fcYOLE on N-Caltech101. The supplied passages identify average precision as the comparison metric and mention a qualitative comparison in the video attachment.
2. Results
- Table 5 compares YOLE and fcYOLE average precision on N-Caltech101 classes.The supplied passages do not state the table's values or which model performs better.Source: §2 · p. 15, Table 5 · p. 16
- A qualitative comparison between YOLE and fcYOLE is also provided in the video attachment.The supplied passage does not describe its outcome.Source: §2 · p. 15
- The table's stated scope is the average-precision comparison between the two models on N-Caltech101.Compare YOLE with fcYOLE without inferring a winner from the caption alone.Source: §2 · p. 15, Table 5 · p. 16