Source-linked AI summary
A Large Scale Event-based Detection Dataset for Automotive
Pierre de Tournemire, Davide Nitti, Etienne Perot, Davide Migliore, Amos Sironi
TL;DR
Event-based vision lacks large, accurately labeled real-world datasets for modern learning and reliable evaluation. This paper releases a large automotive event-camera dataset with diverse recordings and manual car and pedestrian annotations, providing a foundation for detection and related tasks. The dataset is reported as substantially larger than existing event-based datasets in hours and labels.
Problem
Existing event-based datasets remain much smaller and less annotated than frame-based benchmarks, although large accurate datasets are important for reliable vision systems.
Method
The paper constructs a real-world automotive dataset from 39.32 hours across 121 recordings, with manual bounding boxes and train, validation, and test splits separated by recording session.
Results
The dataset is reported as three times larger than DDD17 in hours and has about 22 times more labels than the cited pedestrian dataset.
Takeaways & Limitations
The dataset opens access to large-scale event-camera detection training and is expected to benefit tracking, optical flow, and monocular-depth learning.
Abstract
from arXiv · showhide
We introduce the first very large detection dataset for event cameras. The dataset is composed of more than 39 hours of automotive recordings acquired with a 304x240 ATIS sensor. It contains open roads and very diverse driving scenarios, ranging from urban, highway, suburbs and countryside scenes, as well as different weather and illumination conditions. Manual bounding box annotations of cars and pedestrians contained in the recordings are also provided at a frequency between 1 and 4Hz, yielding more than 255,000 labels in total. We believe that the availability of a labeled dataset of this size will contribute to major advances in event-based vision tasks such as object detection and classification. We also expect benefits in other tasks such as optical flow, structure from motion and tracking, where for example, the large amount of data can be leveraged by self-supervised learning methods.
1. Introduction
Event-based vision needs larger real-world datasets because existing labeled resources are much smaller than frame-based datasets, while real sequences capture sensor properties and imperfections that simulations cannot fully reproduce. This work releases a large automotive dataset with extensive recordings and manual multi-class detection annotations.
- Motivation: Event-based labeled datasets remain much smaller than frame-based datasets, limiting benchmark scale and data availability.The largest labeled event-based classification dataset cited has 100,800 samples, compared with 14 million labeled images in Imagenet.
- Motivation: Real event sequences are needed to capture event-camera properties, noise, and sensor unidealities that are difficult to obtain from frames or idealized simulations.The paper specifically highlights sensor behavior that cannot be fully reproduced by frame-based data or idealized simulation models.
- Contribution: The dataset provides more than 39 hours of realistic automotive recordings from a GEN1 event camera and 255,781 manually labeled car and pedestrian bounding boxes.Annotations comprise 228,123 cars and 27,658 pedestrians, collected at 1Hz, 2Hz, or 4Hz depending on the sequence.
- Contribution: To the authors’ knowledge, it is the largest released event-based dataset by both total hours and total labels, and the only automotive dataset with accurate multi-class detection localization.The annotations cover cars and pedestrians in realistic driving conditions.
- Expected impact: The dataset is intended to narrow the gap between frame-based and event-based datasets and support supervised detection and classification alongside self-supervised optical flow, depth, and tracking.The paper presents these as expected benefits rather than measured outcomes.
2. Related Work
Prior event-based datasets span recognition, detection, geometry, and motion tasks, but many are converted, simulated, narrowly scoped, or small. The proposed dataset addresses this landscape with large-scale real driving data and accurate manual detection annotations.
- Recognition datasets: Earlier recognition datasets often converted frame-based data into events, producing large but unnatural and constrained sequences.Screen-based conversion avoids costly manual labeling but limits event realism through planar displays and screen frequency.
- Recognition datasets: Real recognition datasets improved realism, but their scale or temporal coverage remained limited, including 100,800 short gesture samples representing less than three hours.Each gesture sample contains only 100ms cropped from longer sequences.
- Detection datasets: The first real event-based detection dataset used 12 surveillance sequences of 30 seconds, whereas this work targets large-scale automotive detection.The earlier dataset included pedestrian detection, action recognition, and fall detection.
- Other tasks: Other datasets supported driving, depth, optical flow, stereo, pose, motion segmentation, and corner detection through complementary sensors or specialized acquisition setups.Examples include GPS and vehicle data, lidar depth, pose ground truth, and stereo recordings.
- Simulation versus real data: Simulation offers a valid complement to real collection, but real data remains essential for high dynamic range, high temporal resolution, noise, sensor unidealities, and read-out effects.These properties are difficult to reproduce accurately with idealized simulation models.
- Contribution: Available event-based datasets remain much smaller and less annotated than frame-based benchmarks, motivating this first large automotive detection dataset with accurate manual car and pedestrian labels.The paper positions its dataset as containing more than 39 hours and as the largest event-based dataset publicly available.
3. The ATIS Automotive Detection Dataset
The dataset uses a GEN1 304x240 event camera to collect diverse real-world driving recordings and manually aligned car and pedestrian bounding boxes. It publicly provides event data, annotations, evaluation support, and train/validation/test splits.
- Data collection: 39.32 hours across 121 recordings were collected on varied French roads, scenarios, times, weather, and illumination conditions.Recordings lasted from tens of minutes to several consecutive hours and produced about 750GB of uncompressed raw event data.
- Labeling Protocol: Shared pixels and precise timestamps let image bounding boxes serve directly as ground truth for the event stream without calibration or rectification.Each gray-level image is associated with the timestamp of the last measurement used to generate it.
- Labeling Protocol: Objects smaller than 30 pixels were discarded, and objects were annotated under specified class and visibility rules.Cars exclude buses, trucks, large vehicles, motorbikes, and two-wheelers; pedestrians include skateboarders and kick-scooter users but exclude people inside cars or buildings.
- Labeling Protocol: 228,123 cars and 27,658 pedestrians were manually annotated at 1Hz, 2Hz, or 4Hz.Annotations were drawn on gray-level images generated from the GEN1 sensor’s luminous-intensity measurements.
- Dataset Format and Download: Recordings were split into train, validation, and test sets by keeping each recording session within a single split.For training, continuous recordings were cut into 60-second chunks: 1460 train, 470 test, and 429 validation samples.
- Dataset Format and Download: The public release includes binary event files, NumPy bounding-box annotations, sample visualization code, and a COCO-style evaluation wrapper.Gray-level measurements are not included in the distributed dataset.
4. Analysis and Statistics
Statistics show broad variation in event rates, object geometry, and spatial occurrence across the dataset. Compared with prior event-based datasets, GEN1 is substantially larger in hours and detection labels.
- Event-stream statistics: Most 1ms windows have event rates below 200Kev/s, while unfiltered peaks reach 3Mev/s.The distribution has a long tail associated with strong lighting changes.
- Bounding-box statistics: Car bounding-box heatmaps show two main horizontal camera-position patterns and more boxes on the image’s right side.The right-side concentration is attributed to driving mostly in the right lane; the camera-position pattern is less visible for pedestrians.
- Bounding-box statistics: Pedestrian aspect ratios are approximately Gaussian with mean 0.35, whereas car aspect ratios are closer to bimodal.Car aspect ratios differ between front or rear views and side views.
- Bounding-box statistics: Both object classes have long-tailed bounding-box diagonal distributions beginning at the 30-pixel annotation threshold.Train, validation, and test splits have similar statistics.
- Dataset comparison: GEN1 is 3 times larger than DDD17 in hours and has about 22 times more labels than the prior pedestrian dataset.The second-largest comparison dataset has approximately 2.5 less labels and addresses classification with 100ms samples.
5. Conclusion
The paper presents a large automotive event-camera detection dataset intended to support deep-learning research. It also anticipates benefits for tracking and unsupervised optical-flow and monocular-depth learning.
- Conclusion: The work presents the first large automotive dataset for detection with event cameras.The authors state that the dataset opens the way to training deep-learning detection models on event-based cameras.
- Conclusion: The dataset is expected to benefit object tracking and unsupervised learning of optical flow and monocular depth.These applications are listed alongside detection as intended uses of the released data.
- Conclusion: The authors hope the dataset becomes a reference benchmark and contributes to considerable advances in event-based vision accuracy.This is presented as an expectation about the research community’s use of the dataset.