Source-linked AI summary
HATS: Histograms of Averaged Time Surfaces for Robust Event-based Object Classification
Amos Sironi, Manuele Brambilla, Nicolas Bourdis, Xavier Lagorce, Ryad Benosman
TL;DR
Event-based object classification is constrained by limited feature and architecture designs and by the lack of large real-world datasets. The paper introduces HATS, using local memory to leverage past temporal information, and releases a large real-world car dataset. Across five datasets, it reports higher classification rates and faster computation than existing event-based algorithms.
Problem
Event-based object classification lacks effective low-level representations and architectures, while large real-world event-based datasets remain scarce.
Method
The paper introduces HATS, a representation built from Local Memory Time Surfaces and locally shared memory units, and presents a real-world car-classification dataset.
Results
The method achieves higher classification rates and faster computation times than existing event-based algorithms on the new dataset and four other challenging datasets.
Takeaways & Limitations
HATS supports event-based object recognition by using past-event information with local memory units, while the dataset expands real-world evaluation for car classification.
Takeaways & Limitations
Previously available datasets were generated by displaying images on a monitor, limiting their representativeness because object motion was predefined.
Abstract
from arXiv · showhide
Event-based cameras have recently drawn the attention of the Computer Vision community thanks to their advantages in terms of high temporal resolution, low power consumption and high dynamic range, compared to traditional frame-based cameras. These properties make event-based cameras an ideal choice for autonomous vehicles, robot navigation or UAV vision, among others. However, the accuracy of event-based object classification algorithms, which is of crucial importance for any reliable system working in real-world conditions, is still far behind their frame-based counterparts. Two main reasons for this performance gap are: 1. The lack of effective low-level representations and architectures for event-based object classification and 2. The absence of large real-world event-based datasets. In this paper we address both problems. First, we introduce a novel event-based feature representation together with a new machine learning architecture. Compared to previous approaches, we use local memory units to efficiently leverage past temporal information and build a robust event-based representation. Second, we release the first large real-world event-based dataset for object classification. We compare our method to the state-of-the-art with extensive experiments, showing better classification performance and real-time computation.
1. Introduction
Event-based object classification remains limited by inadequate feature representations and architectures, plus a shortage of large real-world datasets. The paper addresses both gaps with HATS and a new real-world car dataset, reporting higher accuracy and faster computation than existing event-based methods.
- Motivation: Event-based systems offer high temporal resolution, low power consumption, and high dynamic range, but their real-world performance remains inferior to frame-based counterparts.These properties support applications constrained by latency, power, or bandwidth, while the field's performance gap remains unresolved.
- Problem: Two main limitations are insufficient low-level representations and architectures for event-based classification and a lack of large real-world event-based datasets.The paper frames both limitations as targets for improvement.
- Method: HATS combines Local Memory Time Surfaces with locally shared memory units to use past events efficiently and build a robust event-based representation.Neighboring pixels share memory blocks, allowing Local Memory Time Surfaces to be combined into a higher-order representation.
- Dataset: The authors introduce a real-world car-classification dataset containing about 24k samples collected in urban and motorway environments.The samples were semi-automatically annotated, and the authors describe the dataset as the largest labeled real-world event-based dataset to their knowledge.
- Results: The method achieves higher classification rates and faster computation times than existing event-based algorithms across the new dataset and four other challenging datasets.The evaluation compares the proposed method with existing event-based approaches.
2. Event-based camera
Event-based cameras encode visual changes asynchronously rather than producing dense frames at a fixed rate. The resulting event stream carries pixel location, microsecond timing, and polarity, while this classification pipeline uses change-events only.
- Representation: Conventional cameras encode scenes as dense information sampled at a fixed frame rate, whereas event-based cameras adapt encoding to scene content.Event-based sensing is presented as an alternative representation designed to encode observed scenes adaptively.
- Event generation: ATIS pixels asynchronously emit events when they detect sufficient local illuminance change.Each event includes the pixel's x-y position, the microsecond timestamp of the change, and its polarity.
- Pipeline input: The classification pipeline operates on change-events only, while ATIS luminous-intensity measurements are used to generate dataset ground-truth annotations.This design supports event-based cameras regardless of whether they provide gray-level information.
3. Related work
Related work spans frame-based feature learning, event-based representations and classifiers, and dataset limitations. The paper positions its contribution around more suitable low-level representations and larger real-world evaluation data.
- Frame-based features: Frame-based computer vision shifted toward data-driven feature learning as large datasets and effective learning algorithms became available.
- Event-based features: Event-based vision remains an early-stage field, with no clear consensus on the feature representations and architectures best suited to its problems.
- Event-based features: Prior event-based feature work largely targeted stable-feature detection and tracking, including corners, edges, lines, optical flow, and gesture recognition.
- Event-based classification: Spiking neural networks are common for event cameras, but discrete events make gradient-descent training difficult; smoothing or conversion approaches remain suboptimal or hard to scale.
- Paper's positioning: The paper addresses these gaps with a memory-based, temporally and spatially regularized representation and a labeled real-world dataset intended to support robust event-based classification.
- Event-based datasets: Existing event-based classification datasets were often small, converted from standard videos or images, or lacked object-recognition labels, limiting real-world evaluation.
4. Method
The method builds HATS by computing local-memory time surfaces from event histories, aggregating them into cell histograms, and concatenating normalized descriptors. Shared local memory units reduce inefficient neighborhood access while preserving past temporal information for robust, efficient classification.
- Time Surfaces: A time surface encodes local spatiotemporal activity around an event using neighboring-event arrival times and a temporal decay factor.The operator provides both temporal and spatial information for classifying patterns.
- Local Memory Time Surfaces: The local memory time surface incorporates events from a temporal window Δt instead of only the most recent neighboring event.This formulation is described as more robust to noise and small event-stream variations, with better classification accuracy than the standard formulation.
- Local Memory Time Surfaces: Past events are stored in cell-level memory units, allowing neighboring pixels to share information without significantly increasing memory requirements.For each event, the time surface is computed from the cell memory, the histogram is updated, and the event is then added to memory.
- Histograms of Averaged Time Surfaces: HATS aggregates time-surface components within K × K cells into histograms and concatenates the resulting cell descriptors across positions, polarities, and cells.The histograms are normalized by the number of events in the spatiotemporal window to improve contrast invariance.
- Efficient Computation: Shared cell memory avoids scanning the entire temporally ordered event stream because events in the same cell share most of the neighborhood needed for local-memory computation.The architecture addresses inefficient memory access caused by sparse, asynchronous events and can be parallelized for neuromorphic hardware.
5. Datasets
The paper evaluates event-based classification on four frame-derived datasets and introduces N-CARS, a larger real-world dataset recorded directly in urban driving environments. N-CARS includes varied poses, speeds, occlusions, and backgrounds, addressing the limited motion realism of monitor-based conversions.
- Dataset Evaluation: The approach is validated on five datasets: N-MNIST, N-Caltech101, MNIST-DVS, CIFAR10-DVS, and the real-world N-CARS dataset.The four legacy datasets convert standard frame-based datasets into event streams, while N-CARS was recorded from real-world scenes and publicly released.
- Frame-Derived Datasets: The four existing datasets were created by displaying images or moving images on monitors and recording them with event-based sensors.Their event streams are conversions of the original image datasets for evaluating event-based object classification.
- Dataset Limitations: Monitor-generated datasets are not representative of real-world situations because object motion is limited and predefined.They remain useful for initial evaluation of event-based classifiers.
- N-CARS: N-CARS was recorded directly in urban environments using an event-based sensor mounted behind a car windshield and annotated through a semi-automatic protocol.The dataset contains 12,336 car samples and 11,693 non-car samples extracted from synchronized event and grayscale measurements.
- N-CARS: N-CARS contains cars at different poses, speeds, and occlusions across a wide variety of background scenarios, making it a challenging real-world dataset.The paper describes it as the largest labeled event-based dataset acquired in real-world conditions at the time of publication.
6. Experiments
Experiments compare HATS with established event-based methods across frame-converted datasets and real-world N-CARS, emphasizing accuracy, computation, and latency. HATS achieves strong classification performance while operating substantially faster than the compared baselines.
- Experimental setup: A linear SVM is used throughout to isolate the contribution of the event-based feature representation.More complex classifiers could further improve the results.
- Experimental setup: The comparison includes HOTS, H-First, SNN, and a two-layer Gabor-SNN implementation where published SNN results were unavailable.HOTS is evaluated with a linear SVM, favoring that baseline relative to its original classifier.
- Classification accuracy: HATS reports the highest classification rate among event-based methods on datasets converted from frames, with larger improvements on N-Caltech101 and CIFAR10-DVS.HOTS and predefined Gabor-SNN have similar performance, while H-First does not reach good performance.
- Classification accuracy: On N-CARS, HATS outperforms the baselines by a large margin, while spatio-temporal regularization is described as more robust to dataset noise and variability.The N-CARS evaluation reports both global accuracy and AUC.
- Computational time: HATS is more than 20x faster than HOTS, almost 40x faster than a two-layer SNN, and 13 times faster than real time.Computational time is measured per sample, with all methods implemented in C++ and run on the same laptop.
- Latency: Using only 10ms of events, HATS outperforms baselines evaluated on the full 100ms stream, illustrating the latency–accuracy trade-off.Longer windows provide more information but increase latency and can mix dynamics from separate objects.
7. Conclusion and Future Work
The paper presents HATS as a feature representation for event-based object recognition, using past-event information to improve efficiency and accuracy. It also identifies replacing raw events in memory units with learned feature representations as future work.
- HATS represents event-based object-recognition information using histograms of averaged time surfaces.
- More than 13 times faster than real time on N-CARS, while HOTS and Gabor-SNN are respectively 1,5 and 2,8 times slower than real time.The comparison uses average computational time per 100ms sample.
- The architecture uses local memory units shared by neighboring pixels to efficiently exploit past information.
- The proposed architecture outperforms existing spike-based methods in both accuracy and efficiency.
- Future work will extend the method by using a feature representation in memory units instead of raw events.